[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:57 dist-test-slave-kr25 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:57 dist-test-slave-kr25 krb5kdc[395](info): set up 2 sockets
Sep 27 22:46:57 dist-test-slave-kr25 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 22:46:58.470860 388 mini_kdc.cc:121] Time spent starting KDC: real 2.194s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 22:46:58.662796 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.191s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 22:46:58.894573 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.153s user 0.000s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/test-user.keytab.
W20260927 22:46:59.371595 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.477s user 0.000s sys 0.004s
Sep 27 22:46:59 dist-test-slave-kr25 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260927 22:47:00.086962 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.675s user 0.000s sys 0.006s
I20260927 22:47:00.088493 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42145
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:00.407241 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:00.407553 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:00.407684 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:00.414921 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:00.415035 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:00.415094 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:00.415176 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:00.415237 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:00.425612 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:00.426942 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:00.428334 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:00.439703 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:00.439805 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:00.439883 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:00.440876 408 server_base.cc:1034] running on GCE node
I20260927 22:47:00.441594 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:00.442960 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:00.444136 408 hybrid_clock.cc:648] HybridClock initialized: now 1790549220444056 us; error 72 us; skew 500 ppm
I20260927 22:47:00.444473 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:00 dist-test-slave-kr25 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:00.447283 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:00.448602 408 webserver.cc:460] Webserver started at http://127.0.97.62:43625/ using document root <none> and password file <none>
I20260927 22:47:00.449234 408 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:00.449350 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:00.449609 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:00.452472 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "87b09e0cba7d42e08650be7d65230744"
format_stamp: "Formatted at 2026-09-27 22:47:00 on dist-test-slave-kr25"
server_key: "1c8804bc0209ca0659d20ea17932d427"
server_key_iv: "c1a4c5fcb903f4044c28807577edc45e"
server_key_version: "encryptionkey@0"
I20260927 22:47:00.453166 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "87b09e0cba7d42e08650be7d65230744"
format_stamp: "Formatted at 2026-09-27 22:47:00 on dist-test-slave-kr25"
server_key: "1c8804bc0209ca0659d20ea17932d427"
server_key_iv: "c1a4c5fcb903f4044c28807577edc45e"
server_key_version: "encryptionkey@0"
I20260927 22:47:00.457990 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:00.461512 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:00.462792 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:47:00.462940 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "87b09e0cba7d42e08650be7d65230744"
format_stamp: "Formatted at 2026-09-27 22:47:00 on dist-test-slave-kr25"
server_key: "1c8804bc0209ca0659d20ea17932d427"
server_key_iv: "c1a4c5fcb903f4044c28807577edc45e"
server_key_version: "encryptionkey@0"
I20260927 22:47:00.463110 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:00.478705 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:00.482440 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:00.482697 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:00.500758 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42145
I20260927 22:47:00.500761 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42145 every 8 connection(s)
I20260927 22:47:00.502102 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:00.503594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 408
I20260927 22:47:00.503847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:00.505630 388 external_mini_cluster.cc:1428] Setting key 36a22e962823e02c73f8248b5318fe0d
I20260927 22:47:00.505853 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:00.516361 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744: Bootstrap starting.
I20260927 22:47:00.520323 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:00.521591 475 log.cc:824] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:00.524317 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744: No bootstrap required, opened a new log
I20260927 22:47:00.532096 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b09e0cba7d42e08650be7d65230744" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42145 } }
I20260927 22:47:00.532455 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:00.532557 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b09e0cba7d42e08650be7d65230744, State: Initialized, Role: FOLLOWER
Sep 27 22:47:00 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:00.533191 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b09e0cba7d42e08650be7d65230744" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42145 } }
I20260927 22:47:00.533401 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:00.533519 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:00.533658 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:00.536665 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b09e0cba7d42e08650be7d65230744" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42145 } }
I20260927 22:47:00.537251 475 leader_election.cc:302] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b09e0cba7d42e08650be7d65230744; no voters:
I20260927 22:47:00.537879 475 leader_election.cc:288] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:00.537976 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:00.540158 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [term 1 LEADER]: Becoming Leader. State: Replica: 87b09e0cba7d42e08650be7d65230744, State: Running, Role: LEADER
I20260927 22:47:00.540467 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:00.540704 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b09e0cba7d42e08650be7d65230744" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42145 } }
I20260927 22:47:00.543356 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b09e0cba7d42e08650be7d65230744. Latest consensus state: current_term: 1 leader_uuid: "87b09e0cba7d42e08650be7d65230744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b09e0cba7d42e08650be7d65230744" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42145 } } }
I20260927 22:47:00.544199 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:00.543936 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b09e0cba7d42e08650be7d65230744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b09e0cba7d42e08650be7d65230744" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42145 } } }
I20260927 22:47:00.544672 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:00.545475 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:00.509147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.0.97.62:42145)
0927 22:47:00.509967 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:00.509985 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:00.511147 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 22:47:00.520211 (+ 9064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:00.520244 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:00.520750 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:00.521162 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:00.523886 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:00.525786 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:00.527406 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:00.528409 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:00.528605 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:00.534252 (+ 5647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:00.534262 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:00.534280 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:00.534334 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:00.538079 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:00.539148 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:00.539158 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:00.539167 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:00.539384 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:00.539980 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:00.539986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:00.539989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:00.540731 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:00.541010 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:00.543649 (+ 2639us) server_negotiation.cc:299] Negotiation successful
0927 22:47:00.544071 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":331,"threads_started":1}
I20260927 22:47:00.551896 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:00.557396 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:00.566397 486 catalog_manager.cc:1347] Generated new cluster ID: 1b62230a7bbc4222aae11b187584d67b
I20260927 22:47:00.566538 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:00.581529 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:00.582712 486 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:00.588445 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b09e0cba7d42e08650be7d65230744: Generated new TSK 0
I20260927 22:47:00.589244 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:00.639977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42145
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:00.976430 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:00.976704 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:00.976827 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:00.983753 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:00.983876 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:00.984036 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:00.993608 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:00.994860 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:00.996167 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:01.009059 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:01.009107 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:01.009166 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:01.010120 501 server_base.cc:1034] running on GCE node
I20260927 22:47:01.010680 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:01.011590 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:01.012797 501 hybrid_clock.cc:648] HybridClock initialized: now 1790549221012735 us; error 56 us; skew 500 ppm
I20260927 22:47:01.013098 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:01 dist-test-slave-kr25 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:01.016489 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:01.017910 501 webserver.cc:460] Webserver started at http://127.0.97.1:42505/ using document root <none> and password file <none>
I20260927 22:47:01.018551 501 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:01.018656 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:01.018941 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:01.021709 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "e3e23fcdfe2e4a949930abd032ab404e"
format_stamp: "Formatted at 2026-09-27 22:47:01 on dist-test-slave-kr25"
server_key: "16cc389f3ae6b7063f917135435de7fe"
server_key_iv: "b0aab7ccca8682c415b791c78b4c303e"
server_key_version: "encryptionkey@0"
I20260927 22:47:01.022445 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e3e23fcdfe2e4a949930abd032ab404e"
format_stamp: "Formatted at 2026-09-27 22:47:01 on dist-test-slave-kr25"
server_key: "16cc389f3ae6b7063f917135435de7fe"
server_key_iv: "b0aab7ccca8682c415b791c78b4c303e"
server_key_version: "encryptionkey@0"
I20260927 22:47:01.027604 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:47:01.031038 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:01.032470 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:01.032615 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "e3e23fcdfe2e4a949930abd032ab404e"
format_stamp: "Formatted at 2026-09-27 22:47:01 on dist-test-slave-kr25"
server_key: "16cc389f3ae6b7063f917135435de7fe"
server_key_iv: "b0aab7ccca8682c415b791c78b4c303e"
server_key_version: "encryptionkey@0"
I20260927 22:47:01.032790 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:01.060607 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:01.064787 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:01.065058 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:01.065997 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:01.067288 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:01.067373 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:01.067507 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:01.067561 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:01.098696 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44669
I20260927 22:47:01.098763 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44669 every 8 connection(s)
I20260927 22:47:01.099933 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:01.106449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 501
I20260927 22:47:01.106657 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:01.107370 388 external_mini_cluster.cc:1428] Setting key 3ce612b510cc9d2c15bb5b1f6977cdd4
Sep 27 22:47:01 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:01.119525 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:01.102203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55437 (local address 127.0.97.62:42145)
0927 22:47:01.102621 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:01.102626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:01.103551 (+ 925us) server_negotiation.cc:407] Connection header received
0927 22:47:01.105316 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:01.105322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:01.105400 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:01.105501 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:01.107948 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.108663 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.110302 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.110919 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.111046 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:01.114387 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:01.114392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:01.114395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:01.114424 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:01.116744 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:01.117381 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:01.117386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:01.117388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:01.117494 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:01.117928 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:01.117932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:01.117934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:01.118179 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:01.118301 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:01.118961 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 22:47:01.119169 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":205,"threads_started":1}
I20260927 22:47:01.120438 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:01.102735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42145 (local address 127.0.97.1:55437)
0927 22:47:01.103383 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:01.103421 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:01.105055 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:01.106269 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:01.106281 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:01.106994 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:01.107773 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:01.107799 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.108812 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.108816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:01.110160 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:01.110172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.111105 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.111109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:01.111309 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:01.112036 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:01.112058 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:01.114192 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:01.116866 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:01.116875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:01.116892 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:01.117266 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:01.117587 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:01.117592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:01.117596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:01.117816 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:01.118293 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:01.118300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:01.118616 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 22:47:01.118834 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 22:47:01.119308 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:01.121825 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42145
I20260927 22:47:01.122104 629 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:01.122766 629 heartbeater.cc:511] Master 127.0.97.62:42145 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:47:01.124760 440 ts_manager.cc:194] Registered new tserver with Master: e3e23fcdfe2e4a949930abd032ab404e (127.0.97.1:44669)
I20260927 22:47:01.126142 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:01.142796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42145
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:01.461210 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:01.461504 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:01.461642 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:01.469365 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:01.469481 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:01.469630 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:01.479794 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:01.481041 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:01.482342 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:01.494805 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:01.495023 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:01.494900 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:01.495811 635 server_base.cc:1034] running on GCE node
I20260927 22:47:01.496361 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:01.497211 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:01.498396 635 hybrid_clock.cc:648] HybridClock initialized: now 1790549221498344 us; error 45 us; skew 500 ppm
I20260927 22:47:01.498677 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:01 dist-test-slave-kr25 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:01.502635 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:01.504421 635 webserver.cc:460] Webserver started at http://127.0.97.2:41167/ using document root <none> and password file <none>
I20260927 22:47:01.505235 635 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:01.505353 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:01.505692 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:01.509508 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "1f839b0feb38449ab81391626d7846a9"
format_stamp: "Formatted at 2026-09-27 22:47:01 on dist-test-slave-kr25"
server_key: "fc802431bb49ab032156caad80275606"
server_key_iv: "7d235e13e4f67093ae0133b5c15a6eb6"
server_key_version: "encryptionkey@0"
I20260927 22:47:01.510463 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1f839b0feb38449ab81391626d7846a9"
format_stamp: "Formatted at 2026-09-27 22:47:01 on dist-test-slave-kr25"
server_key: "fc802431bb49ab032156caad80275606"
server_key_iv: "7d235e13e4f67093ae0133b5c15a6eb6"
server_key_version: "encryptionkey@0"
I20260927 22:47:01.516630 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 22:47:01.521330 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:01.523092 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:47:01.523259 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "1f839b0feb38449ab81391626d7846a9"
format_stamp: "Formatted at 2026-09-27 22:47:01 on dist-test-slave-kr25"
server_key: "fc802431bb49ab032156caad80275606"
server_key_iv: "7d235e13e4f67093ae0133b5c15a6eb6"
server_key_version: "encryptionkey@0"
I20260927 22:47:01.523568 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:01.558934 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:01.562906 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:01.563170 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:01.564107 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:01.565791 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:01.565941 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:01.566115 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:01.566190 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:01.600415 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46275
I20260927 22:47:01.600476 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46275 every 8 connection(s)
I20260927 22:47:01.601722 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:01.607585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 635
I20260927 22:47:01.607765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:01.608150 388 external_mini_cluster.cc:1428] Setting key d6aa0e1b916381290b7ce087aa0d7c2c
Sep 27 22:47:01 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:01.623461 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:01.604328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43139 (local address 127.0.97.62:42145)
0927 22:47:01.604490 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:01.604494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:01.605586 (+ 1092us) server_negotiation.cc:407] Connection header received
0927 22:47:01.607352 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:01.607357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:01.607454 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:01.607542 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:01.609131 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.610111 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.612387 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.613248 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.613353 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:01.617113 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:01.617120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:01.617125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:01.617160 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:01.620237 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:01.621034 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:01.621039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:01.621042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:01.621187 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:01.621645 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:01.621650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:01.621654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:01.621977 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:01.622122 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:01.622863 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 22:47:01.623077 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:47:01.624444 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:01.604735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42145 (local address 127.0.97.2:43139)
0927 22:47:01.605437 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:01.605477 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:01.607081 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:01.607803 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:01.607812 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:01.608350 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:01.608950 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:01.608974 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.610251 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.610256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:01.612251 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:01.612265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:01.613565 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:01.613573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:01.613901 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:01.614606 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:01.614640 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:01.616880 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:01.620397 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:01.620405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:01.620419 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:01.620918 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:01.621309 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:01.621314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:01.621318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:01.621541 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:01.622090 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:01.622095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:01.622435 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 22:47:01.622740 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 22:47:01.623140 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":262,"threads_started":1}
I20260927 22:47:01.625936 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42145
I20260927 22:47:01.626279 763 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:47:01.627084 763 heartbeater.cc:511] Master 127.0.97.62:42145 requested a full tablet report, sending...
I20260927 22:47:01.628724 440 ts_manager.cc:194] Registered new tserver with Master: 1f839b0feb38449ab81391626d7846a9 (127.0.97.2:46275)
I20260927 22:47:01.629488 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:01.645972 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42145
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:01.970490 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:01.970754 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:01.970881 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:01.978013 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:01.978125 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:01.978289 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:01.988473 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:01.989701 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:01.990962 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:02.003245 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:02.003316 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:02.003343 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:02.004129 768 server_base.cc:1034] running on GCE node
I20260927 22:47:02.004789 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:02.005764 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:02.006968 768 hybrid_clock.cc:648] HybridClock initialized: now 1790549222006911 us; error 43 us; skew 500 ppm
I20260927 22:47:02.007321 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:02.010730 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:02.012164 768 webserver.cc:460] Webserver started at http://127.0.97.3:38639/ using document root <none> and password file <none>
I20260927 22:47:02.012799 768 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:02.012894 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:02.013157 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:02.015974 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "27358592a4b54158bb9dfff65b0b633a"
format_stamp: "Formatted at 2026-09-27 22:47:02 on dist-test-slave-kr25"
server_key: "fc534f2acb54e55f875d7828cb11e15e"
server_key_iv: "7d9f80cdf9fe0f41389c4cab3f3ed1ce"
server_key_version: "encryptionkey@0"
I20260927 22:47:02.016672 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "27358592a4b54158bb9dfff65b0b633a"
format_stamp: "Formatted at 2026-09-27 22:47:02 on dist-test-slave-kr25"
server_key: "fc534f2acb54e55f875d7828cb11e15e"
server_key_iv: "7d9f80cdf9fe0f41389c4cab3f3ed1ce"
server_key_version: "encryptionkey@0"
I20260927 22:47:02.021582 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:47:02.024715 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:02.025988 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:47:02.026116 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "27358592a4b54158bb9dfff65b0b633a"
format_stamp: "Formatted at 2026-09-27 22:47:02 on dist-test-slave-kr25"
server_key: "fc534f2acb54e55f875d7828cb11e15e"
server_key_iv: "7d9f80cdf9fe0f41389c4cab3f3ed1ce"
server_key_version: "encryptionkey@0"
I20260927 22:47:02.026273 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:02.039072 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:02.043586 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:02.043870 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:02.044739 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:02.046196 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:02.046283 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:02.046401 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:02.046460 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:02.077834 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38103
I20260927 22:47:02.077879 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38103 every 8 connection(s)
I20260927 22:47:02.079034 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:02.079691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 768
I20260927 22:47:02.079877 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:02.080376 388 external_mini_cluster.cc:1428] Setting key d6796500e17ecf75ad775202e13bcb74
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:02.098929 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:02.081727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47113 (local address 127.0.97.62:42145)
0927 22:47:02.081902 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:02.081908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:02.083305 (+ 1397us) server_negotiation.cc:407] Connection header received
0927 22:47:02.085673 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:02.085679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:02.085772 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:02.085854 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:02.087360 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.088152 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.089674 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.090058 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.090173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:02.093457 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:02.093462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:02.093465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:02.093495 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:02.096166 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.096884 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.096888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.096890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.097010 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.097333 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.097336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.097338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.097598 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:02.097699 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:02.098386 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 22:47:02.098623 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:47:02.099737 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:02.082404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42145 (local address 127.0.97.3:47113)
0927 22:47:02.083162 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:02.083200 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:02.085412 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:02.086109 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:02.086118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:02.086638 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:02.087202 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:02.087219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.088297 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.088301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:02.089528 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:02.089540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.090263 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.090269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:02.090495 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:02.091102 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:02.091123 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:02.093214 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:02.096299 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:02.096307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:02.096323 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:02.096778 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:02.097086 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:02.097089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:02.097091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:02.097259 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:02.097684 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:02.097689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:02.098009 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:47:02.098237 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 22:47:02.098643 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":319,"threads_started":1}
I20260927 22:47:02.100926 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42145
I20260927 22:47:02.101184 896 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:02.101876 896 heartbeater.cc:511] Master 127.0.97.62:42145 requested a full tablet report, sending...
I20260927 22:47:02.103149 440 ts_manager.cc:194] Registered new tserver with Master: 27358592a4b54158bb9dfff65b0b633a (127.0.97.3:38103)
I20260927 22:47:02.103931 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47113
I20260927 22:47:02.108493 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260927 22:47:02.129307 629 heartbeater.cc:503] Master 127.0.97.62:42145 was elected leader, sending a full tablet report...
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:02.152843 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:02.135105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.0.97.1:44669)
0927 22:47:02.135674 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:02.135681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:02.135805 (+ 124us) server_negotiation.cc:407] Connection header received
0927 22:47:02.135971 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:02.135977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:02.136178 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:02.136306 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:02.137570 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.138583 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.139884 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.140471 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.140561 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:02.144226 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:02.144237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:02.144251 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:02.144283 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:02.150030 (+ 5747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.151063 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.151070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.151073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.151195 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.151539 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.151542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.151544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.151854 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:02.152082 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:02.152312 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 22:47:02.152498 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":243,"threads_started":1}
W20260927 22:47:02.154122 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42116
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:02.191866 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:02.179277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.0.97.1:44669)
0927 22:47:02.179513 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:02.179519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:02.179793 (+ 274us) server_negotiation.cc:407] Connection header received
0927 22:47:02.179966 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:02.179970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:02.180059 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:02.180145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:02.181425 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.182098 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.183264 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.183696 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.183804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:02.187316 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:02.187321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:02.187324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:02.187356 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:02.189611 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.190178 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.190183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.190185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.190300 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.190604 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.190609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.190612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.190842 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:02.191051 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:02.191332 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:47:02.191569 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:47:02.193214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 501
I20260927 22:47:02.204499 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:02.194099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.0.97.1:44669)
0927 22:47:02.194243 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:02.194247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:02.194509 (+ 262us) server_negotiation.cc:407] Connection header received
0927 22:47:02.194631 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:02.194634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:02.194685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:02.194752 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:02.196248 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.196904 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.198109 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.198518 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.198607 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:02.200519 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:02.200524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:02.200527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:02.200553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:02.202624 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.203126 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.203130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.203132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.203236 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.203529 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.203532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.203534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.203751 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:02.203908 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:02.204116 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:47:02.204247 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:47:02.205169 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:02.285506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 501
I20260927 22:47:02.298617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 635
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:02.313230 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:02.299529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.0.97.2:46275)
0927 22:47:02.299994 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:02.300001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:02.300024 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:47:02.300080 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:02.300086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:02.300279 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:02.300426 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:02.301596 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.302591 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.303921 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.304466 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.304577 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:02.308332 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:02.308343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:02.308357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:02.308394 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:02.310969 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.311605 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.311612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.311614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.311729 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.312041 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.312043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.312045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.312348 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:02.312566 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:02.312731 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 22:47:02.312918 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
I20260927 22:47:02.314486 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:02.394913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 635
I20260927 22:47:02.413219 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 768
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:02.428541 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:02.414230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.0.97.3:38103)
0927 22:47:02.414843 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:02.414850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:02.414879 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:47:02.414935 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:02.414942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:02.415157 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:02.415355 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:02.416535 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.417552 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.418766 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:02.419316 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:02.419425 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:02.423236 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:02.423246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:02.423269 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:02.423301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:02.425875 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.426480 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.426488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.426490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.426605 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:02.426943 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:02.426947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:02.426949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:02.427290 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:02.427583 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:02.427967 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 22:47:02.428210 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":297,"threads_started":1}
I20260927 22:47:02.429759 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:02.509500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 768
I20260927 22:47:02.522230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 408
I20260927 22:47:02.523411 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:02.620851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 408
2026-09-27T22:47:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6390 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[917](info): set up 2 sockets
Sep 27 22:47:02 dist-test-slave-kr25 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 22:47:03.599182 388 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:03 dist-test-slave-kr25 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:03.752374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40871
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:04.049227 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:04.049495 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:04.049606 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:04.056630 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:04.056722 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:04.056800 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:04.056885 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:04.056946 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:04.066208 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:04.067502 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:04.068794 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:04.083510 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:04.083636 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:04.083667 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:04.084671 930 server_base.cc:1034] running on GCE node
I20260927 22:47:04.085453 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:04.086803 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:04.087993 930 hybrid_clock.cc:648] HybridClock initialized: now 1790549224087948 us; error 39 us; skew 500 ppm
I20260927 22:47:04.088302 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:04 dist-test-slave-kr25 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:04.091180 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:04.092614 930 webserver.cc:460] Webserver started at http://127.0.97.62:32985/ using document root <none> and password file <none>
I20260927 22:47:04.093271 930 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:04.093376 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:04.093638 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:04.096422 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "3c8f2038d14e4b44970adead059432bf"
format_stamp: "Formatted at 2026-09-27 22:47:04 on dist-test-slave-kr25"
server_key: "2bed37d0ebbcdd5aa4b0fb6b1e7b8a4b"
server_key_iv: "e4121c6ba06a072c7efa4528690aac78"
server_key_version: "encryptionkey@0"
I20260927 22:47:04.097116 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c8f2038d14e4b44970adead059432bf"
format_stamp: "Formatted at 2026-09-27 22:47:04 on dist-test-slave-kr25"
server_key: "2bed37d0ebbcdd5aa4b0fb6b1e7b8a4b"
server_key_iv: "e4121c6ba06a072c7efa4528690aac78"
server_key_version: "encryptionkey@0"
I20260927 22:47:04.101866 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:04.104966 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:04.106190 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:47:04.106335 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "3c8f2038d14e4b44970adead059432bf"
format_stamp: "Formatted at 2026-09-27 22:47:04 on dist-test-slave-kr25"
server_key: "2bed37d0ebbcdd5aa4b0fb6b1e7b8a4b"
server_key_iv: "e4121c6ba06a072c7efa4528690aac78"
server_key_version: "encryptionkey@0"
I20260927 22:47:04.106499 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:04.126932 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:04.130589 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:04.130831 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:04.153439 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40871
I20260927 22:47:04.153484 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40871 every 8 connection(s)
I20260927 22:47:04.154819 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:04.156458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 930
I20260927 22:47:04.156673 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:04.157473 388 external_mini_cluster.cc:1428] Setting key 01c71dfac196f7708e9ad1413451a061
I20260927 22:47:04.159307 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:47:04 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:04.172048 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf: Bootstrap starting.
I20260927 22:47:04.175048 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:04.176225 997 log.cc:824] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:04.178622 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:04.159386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.0.97.62:40871)
0927 22:47:04.160119 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:04.160130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:04.160169 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:47:04.161686 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:04.161706 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:04.162097 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:04.162410 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:04.164045 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:04.165302 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:04.166940 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:04.167616 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:04.167734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:04.172128 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:04.172135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:04.172149 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:04.172192 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:04.174876 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:04.175605 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:04.175611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:04.175618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:04.175776 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:04.176169 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:04.176172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:04.176174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:04.176731 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:04.177044 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:04.177393 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 22:47:04.177724 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":319,"threads_started":1}
I20260927 22:47:04.178987 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf: No bootstrap required, opened a new log
I20260927 22:47:04.186281 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f2038d14e4b44970adead059432bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40871 } }
I20260927 22:47:04.186612 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:04.186702 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8f2038d14e4b44970adead059432bf, State: Initialized, Role: FOLLOWER
I20260927 22:47:04.187192 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f2038d14e4b44970adead059432bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40871 } }
I20260927 22:47:04.187399 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:04.187537 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:04.187686 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:04.189633 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f2038d14e4b44970adead059432bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40871 } }
I20260927 22:47:04.190073 997 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8f2038d14e4b44970adead059432bf; no voters:
I20260927 22:47:04.190585 997 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:04.190687 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:04.191284 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [term 1 LEADER]: Becoming Leader. State: Replica: 3c8f2038d14e4b44970adead059432bf, State: Running, Role: LEADER
I20260927 22:47:04.191890 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f2038d14e4b44970adead059432bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40871 } }
I20260927 22:47:04.192581 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:04.194518 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8f2038d14e4b44970adead059432bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f2038d14e4b44970adead059432bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40871 } } }
I20260927 22:47:04.194481 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8f2038d14e4b44970adead059432bf. Latest consensus state: current_term: 1 leader_uuid: "3c8f2038d14e4b44970adead059432bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f2038d14e4b44970adead059432bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40871 } } }
I20260927 22:47:04.194976 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:04.194972 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:04.196779 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:04.201706 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:04.209465 1008 catalog_manager.cc:1347] Generated new cluster ID: 294af74ce934448ab680567d2857041d
I20260927 22:47:04.209571 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:04.227670 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:04.228915 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:04.236093 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8f2038d14e4b44970adead059432bf: Generated new TSK 0
I20260927 22:47:04.236840 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:04.283610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40871
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:04.585891 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:04.586166 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:04.586285 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:04.593415 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:04.593519 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:04.593667 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:04.603101 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:04.604432 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:04.605736 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:04.617619 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:04.617738 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:04.617769 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:04.618458 1023 server_base.cc:1034] running on GCE node
I20260927 22:47:04.618944 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:04.619845 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:04.621028 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790549224620978 us; error 46 us; skew 500 ppm
I20260927 22:47:04.621304 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:04 dist-test-slave-kr25 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:04.624228 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:04.625566 1023 webserver.cc:460] Webserver started at http://127.0.97.1:39755/ using document root <none> and password file <none>
I20260927 22:47:04.626178 1023 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:04.626271 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:04.626519 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:04.629316 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "5fd164a51da1481f99cf4c75a3624763"
format_stamp: "Formatted at 2026-09-27 22:47:04 on dist-test-slave-kr25"
server_key: "dddfb409e1c2742c16d35dce317d1126"
server_key_iv: "b3cefcccb88998208ffcc46023c68208"
server_key_version: "encryptionkey@0"
I20260927 22:47:04.629999 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5fd164a51da1481f99cf4c75a3624763"
format_stamp: "Formatted at 2026-09-27 22:47:04 on dist-test-slave-kr25"
server_key: "dddfb409e1c2742c16d35dce317d1126"
server_key_iv: "b3cefcccb88998208ffcc46023c68208"
server_key_version: "encryptionkey@0"
I20260927 22:47:04.634750 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:47:04.637933 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:04.639153 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:04.639277 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "5fd164a51da1481f99cf4c75a3624763"
format_stamp: "Formatted at 2026-09-27 22:47:04 on dist-test-slave-kr25"
server_key: "dddfb409e1c2742c16d35dce317d1126"
server_key_iv: "b3cefcccb88998208ffcc46023c68208"
server_key_version: "encryptionkey@0"
I20260927 22:47:04.639494 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:04.666538 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:04.672158 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:04.672519 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:04.673453 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:04.674762 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:04.674839 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:04.674934 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:04.674981 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:04.702777 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34027
I20260927 22:47:04.702780 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34027 every 8 connection(s)
I20260927 22:47:04.703979 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:04.709048 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 1023
I20260927 22:47:04.709246 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:04.710000 388 external_mini_cluster.cc:1428] Setting key f7f59e23cbe85e063cf977e41b573b0c
Sep 27 22:47:04 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:04.723402 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:04.706220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46527 (local address 127.0.97.62:40871)
0927 22:47:04.706625 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:04.706629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:04.707473 (+ 844us) server_negotiation.cc:407] Connection header received
0927 22:47:04.709485 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:04.709492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:04.709572 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:04.709672 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:04.711410 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:04.712156 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:04.713617 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:04.714007 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:04.714110 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:04.717842 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:04.717847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:04.717850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:04.717879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:04.720323 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:04.721143 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:04.721147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:04.721149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:04.721304 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:04.721795 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:04.721799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:04.721801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:04.722065 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:04.722181 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:04.722980 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 22:47:04.723138 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":217,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:04.724771 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:04.706673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40871 (local address 127.0.97.1:46527)
0927 22:47:04.707315 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:04.707353 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:04.709200 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:04.709938 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:04.709948 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:04.710622 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:04.711255 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:04.711272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:04.712286 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:04.712290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:04.713491 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:04.713502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:04.714205 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:04.714212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:04.714432 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:04.715040 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:04.715072 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:04.717633 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:04.720446 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:04.720462 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:04.720477 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:04.721012 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:04.721412 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:04.721417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:04.721421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:04.721656 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:04.722192 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:04.722200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:04.722602 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 22:47:04.722908 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 22:47:04.723409 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":244,"threads_started":1}
I20260927 22:47:04.726133 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40871
I20260927 22:47:04.726446 1151 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:47:04.727136 1151 heartbeater.cc:511] Master 127.0.97.62:40871 requested a full tablet report, sending...
I20260927 22:47:04.729108 962 ts_manager.cc:194] Registered new tserver with Master: 5fd164a51da1481f99cf4c75a3624763 (127.0.97.1:34027)
I20260927 22:47:04.730485 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:04.744815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40871
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:05.046509 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:05.046798 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:05.046921 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:05.053857 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:05.053951 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:05.054071 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:05.063282 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:05.064481 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:05.065721 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:05.077824 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:05.077980 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:05.078020 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:05.079110 1157 server_base.cc:1034] running on GCE node
I20260927 22:47:05.079699 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:05.080518 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:05.081703 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790549225081658 us; error 41 us; skew 500 ppm
I20260927 22:47:05.081982 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:05.084932 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:05.086333 1157 webserver.cc:460] Webserver started at http://127.0.97.2:35579/ using document root <none> and password file <none>
I20260927 22:47:05.086978 1157 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:05.087101 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:05.087409 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:05.090296 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "ace44b01883147798952c214f6fe4e4f"
format_stamp: "Formatted at 2026-09-27 22:47:05 on dist-test-slave-kr25"
server_key: "83d7142be9ff8b5b3093e5a19bec1494"
server_key_iv: "8cfef66bdf5ab29df8ded1dfbad7bd41"
server_key_version: "encryptionkey@0"
I20260927 22:47:05.091049 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ace44b01883147798952c214f6fe4e4f"
format_stamp: "Formatted at 2026-09-27 22:47:05 on dist-test-slave-kr25"
server_key: "83d7142be9ff8b5b3093e5a19bec1494"
server_key_iv: "8cfef66bdf5ab29df8ded1dfbad7bd41"
server_key_version: "encryptionkey@0"
I20260927 22:47:05.096186 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 22:47:05.099730 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:05.100987 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:47:05.101140 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "ace44b01883147798952c214f6fe4e4f"
format_stamp: "Formatted at 2026-09-27 22:47:05 on dist-test-slave-kr25"
server_key: "83d7142be9ff8b5b3093e5a19bec1494"
server_key_iv: "8cfef66bdf5ab29df8ded1dfbad7bd41"
server_key_version: "encryptionkey@0"
I20260927 22:47:05.101310 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:05.125530 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:05.129513 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:05.129765 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:05.130630 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:05.131917 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:05.131989 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:05.132098 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:05.132148 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:05.163110 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42877
I20260927 22:47:05.163143 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42877 every 8 connection(s)
I20260927 22:47:05.164372 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:05.168718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 1157
I20260927 22:47:05.168881 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:05.169250 388 external_mini_cluster.cc:1428] Setting key a9fd3e01c3d5a1711ab9cf8bb1c63ebe
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:47:05.183549 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.166755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57425 (local address 127.0.97.62:40871)
0927 22:47:05.166909 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.166913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.168220 (+ 1307us) server_negotiation.cc:407] Connection header received
0927 22:47:05.169993 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.169999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.170079 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.170174 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:05.171643 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.172336 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.173822 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.174205 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.174315 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.177700 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:05.177705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:05.177709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:05.177746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:05.180475 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.181200 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.181204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.181206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.181343 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.181781 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.181785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.181787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.182065 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:05.182199 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.182937 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.183155 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:47:05.184530 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.167358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40871 (local address 127.0.97.2:57425)
0927 22:47:05.168076 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:05.168111 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:05.169751 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:05.170395 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:05.170403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:05.170936 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:05.171497 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.171515 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.172471 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.172475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.173683 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.173693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.174407 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.174413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.174627 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.175267 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:05.175287 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:05.177516 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:05.180616 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:05.180626 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:05.180644 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:05.181071 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:05.181454 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:05.181460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:05.181464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:05.181660 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:05.182199 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:05.182208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:05.182561 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 22:47:05.182811 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 22:47:05.183325 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":276,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:47:05.185915 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40871
I20260927 22:47:05.186203 1285 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:05.186941 1285 heartbeater.cc:511] Master 127.0.97.62:40871 requested a full tablet report, sending...
I20260927 22:47:05.188385 962 ts_manager.cc:194] Registered new tserver with Master: ace44b01883147798952c214f6fe4e4f (127.0.97.2:42877)
I20260927 22:47:05.189211 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:05.203080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40871
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:05.505048 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:05.505327 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:05.505434 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:05.512490 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:05.512591 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:05.512712 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:05.522037 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:05.523192 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:05.524490 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:05.536438 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:05.536477 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:05.536502 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:05.537550 1290 server_base.cc:1034] running on GCE node
I20260927 22:47:05.538100 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:05.538914 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:05.540129 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790549225540084 us; error 36 us; skew 500 ppm
I20260927 22:47:05.540445 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:05.543679 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:05.545053 1290 webserver.cc:460] Webserver started at http://127.0.97.3:33547/ using document root <none> and password file <none>
I20260927 22:47:05.545670 1290 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:05.545768 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:05.546064 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:05.548858 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "06932bd982164400acddaef3bf870af6"
format_stamp: "Formatted at 2026-09-27 22:47:05 on dist-test-slave-kr25"
server_key: "1152e74c04bb23d8dbfa6e63539fa965"
server_key_iv: "8d7171f26eed7cbfba83ee958d80a415"
server_key_version: "encryptionkey@0"
I20260927 22:47:05.549577 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "06932bd982164400acddaef3bf870af6"
format_stamp: "Formatted at 2026-09-27 22:47:05 on dist-test-slave-kr25"
server_key: "1152e74c04bb23d8dbfa6e63539fa965"
server_key_iv: "8d7171f26eed7cbfba83ee958d80a415"
server_key_version: "encryptionkey@0"
I20260927 22:47:05.554334 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:05.557698 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:05.559028 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:05.559168 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "06932bd982164400acddaef3bf870af6"
format_stamp: "Formatted at 2026-09-27 22:47:05 on dist-test-slave-kr25"
server_key: "1152e74c04bb23d8dbfa6e63539fa965"
server_key_iv: "8d7171f26eed7cbfba83ee958d80a415"
server_key_version: "encryptionkey@0"
I20260927 22:47:05.559334 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:05.574633 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:05.578334 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:05.578589 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:05.579481 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:05.580739 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:05.580813 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:05.580930 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:05.580981 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:05.610626 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39183
I20260927 22:47:05.610672 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39183 every 8 connection(s)
I20260927 22:47:05.611863 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:05.617305 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 1290
I20260927 22:47:05.617484 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:05.617885 388 external_mini_cluster.cc:1428] Setting key 3b78cd662e9109f2f1d0444979b5834f
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:05.630317 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.614188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43149 (local address 127.0.97.62:40871)
0927 22:47:05.614357 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.614361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.615509 (+ 1148us) server_negotiation.cc:407] Connection header received
0927 22:47:05.617306 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.617310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.617373 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.617485 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:05.618998 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.619824 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.621566 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.621957 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.622046 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.625083 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:05.625088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:05.625091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:05.625122 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:05.627622 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.628332 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.628336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.628338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.628485 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.628866 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.628869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.628877 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.629152 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:05.629296 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.629879 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.630046 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:05.631171 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.614676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40871 (local address 127.0.97.3:43149)
0927 22:47:05.615343 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:05.615387 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:05.617013 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:05.617708 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:05.617716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:05.618221 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:05.618839 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.618857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.619983 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.619988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.621434 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.621446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.622132 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.622136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.622321 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.622886 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:05.622906 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:05.624899 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:05.627727 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:05.627732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:05.627744 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:05.628176 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:05.628579 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:05.628581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:05.628584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:05.628749 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:05.629244 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:05.629250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:05.629536 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 22:47:05.629746 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 22:47:05.630108 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":248,"threads_started":1}
I20260927 22:47:05.632584 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40871
I20260927 22:47:05.632849 1418 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:05.633502 1418 heartbeater.cc:511] Master 127.0.97.62:40871 requested a full tablet report, sending...
I20260927 22:47:05.634764 962 ts_manager.cc:194] Registered new tserver with Master: 06932bd982164400acddaef3bf870af6 (127.0.97.3:39183)
I20260927 22:47:05.635685 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43149
I20260927 22:47:05.644608 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:47:05.662091 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.650697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.97.62:40871)
0927 22:47:05.650876 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.650882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.651203 (+ 321us) server_negotiation.cc:407] Connection header received
0927 22:47:05.651406 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.651411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.651505 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.651584 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:05.652876 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.653555 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.654764 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.655144 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.655244 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.657367 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:05.657373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:05.657377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:05.657407 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:05.659811 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.660368 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.660374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.660377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.660481 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.660794 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.660799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.660803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.661091 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:05.661298 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.661637 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.661835 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:47:05.668715 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:05.675268 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:05.697834 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.687949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.0.97.2:42877)
0927 22:47:05.689925 (+ 1976us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.689934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.689962 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:47:05.690051 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.690056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.690247 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.690438 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:05.691569 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.692936 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.694970 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.696694 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.696823 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.697022 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.697338 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.697513 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1835,"thread_start_us":1727,"threads_started":1}
I20260927 22:47:05.698354 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.686987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34027 (local address 127.0.0.1:33428)
0927 22:47:05.688654 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:05.688670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:05.688767 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:05.690478 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:05.690483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:05.690525 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:05.690945 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.690953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.692771 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.692776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.695365 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.695381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.697502 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.697508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.697663 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.697695 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:47:05.697757 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 22:47:05.697868 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"mutex_wait_us":111,"spinlock_wait_cycles":8320,"thread_start_us":1438,"threads_started":1}
I20260927 22:47:05.698724 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.686314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39183 (local address 127.0.0.1:38890)
0927 22:47:05.687049 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:05.687081 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:05.687249 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:05.690255 (+ 3006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:05.690263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:05.690294 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:05.690756 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.690769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.692578 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.692584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.695587 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.695598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.698098 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.698105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.698224 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.698245 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:47:05.698322 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:47:05.698399 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":348,"thread_start_us":314,"threads_started":1}
I20260927 22:47:05.698886 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.686605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.0.97.3:39183)
0927 22:47:05.689655 (+ 3050us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.689663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.689692 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:47:05.689787 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.689794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.690039 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.690222 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:05.690933 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.692456 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.695733 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.697288 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.697512 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.697777 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.698386 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.698549 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2890,"thread_start_us":2744,"threads_started":1}
I20260927 22:47:05.698032 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.687366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42877 (local address 127.0.0.1:44132)
0927 22:47:05.689310 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:05.689326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:05.689422 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:05.690478 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:05.690483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:05.690517 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:05.691369 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.691380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.693074 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.693078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.694791 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.694805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.696886 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.696893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.697091 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.697147 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:47:05.697294 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:47:05.697526 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1834,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":1726,"threads_started":1}
I20260927 22:47:05.699627 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.687526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.97.1:34027)
0927 22:47:05.689936 (+ 2410us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.689943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.689967 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:47:05.690048 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.690055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.690268 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.690593 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:05.691102 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.692642 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.695542 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.697278 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.699029 (+ 1751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.699177 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.699225 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.699331 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2269,"thread_start_us":2290,"threads_started":1}
I20260927 22:47:05.702903 1353 tablet_service.cc:1467] Processing CreateTablet for tablet df8b1cd836bd4028a38b216c4dd4d2ae (DEFAULT_TABLE table=test-table [id=70bc8f53ddef41b694515d8e0789606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:05.702905 1220 tablet_service.cc:1467] Processing CreateTablet for tablet df8b1cd836bd4028a38b216c4dd4d2ae (DEFAULT_TABLE table=test-table [id=70bc8f53ddef41b694515d8e0789606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:05.702899 1086 tablet_service.cc:1467] Processing CreateTablet for tablet df8b1cd836bd4028a38b216c4dd4d2ae (DEFAULT_TABLE table=test-table [id=70bc8f53ddef41b694515d8e0789606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:05.704244 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8b1cd836bd4028a38b216c4dd4d2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:05.704255 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8b1cd836bd4028a38b216c4dd4d2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:05.705567 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8b1cd836bd4028a38b216c4dd4d2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:05.720088 1436 tablet_bootstrap.cc:492] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763: Bootstrap starting.
I20260927 22:47:05.721786 1437 tablet_bootstrap.cc:492] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6: Bootstrap starting.
I20260927 22:47:05.724080 1436 tablet_bootstrap.cc:654] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:05.724993 1437 tablet_bootstrap.cc:654] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:05.726058 1437 log.cc:824] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:05.726058 1436 log.cc:824] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:05.729861 1438 tablet_bootstrap.cc:492] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f: Bootstrap starting.
I20260927 22:47:05.730623 1436 tablet_bootstrap.cc:492] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763: No bootstrap required, opened a new log
I20260927 22:47:05.730667 1437 tablet_bootstrap.cc:492] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6: No bootstrap required, opened a new log
I20260927 22:47:05.730899 1436 ts_tablet_manager.cc:1397] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 22:47:05.730912 1437 ts_tablet_manager.cc:1397] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 22:47:05.733261 1151 heartbeater.cc:503] Master 127.0.97.62:40871 was elected leader, sending a full tablet report...
I20260927 22:47:05.734151 1438 tablet_bootstrap.cc:654] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:05.735347 1438 log.cc:824] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:05.740020 1436 raft_consensus.cc:348] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.740327 1436 raft_consensus.cc:374] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:05.740417 1436 raft_consensus.cc:729] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd164a51da1481f99cf4c75a3624763, State: Initialized, Role: FOLLOWER
I20260927 22:47:05.740567 1437 raft_consensus.cc:348] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.740960 1437 raft_consensus.cc:374] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:05.741076 1438 tablet_bootstrap.cc:492] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f: No bootstrap required, opened a new log
I20260927 22:47:05.741062 1437 raft_consensus.cc:729] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06932bd982164400acddaef3bf870af6, State: Initialized, Role: FOLLOWER
I20260927 22:47:05.740937 1436 consensus_queue.cc:260] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.741323 1438 ts_tablet_manager.cc:1397] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260927 22:47:05.741592 1437 consensus_queue.cc:260] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.742614 1418 heartbeater.cc:503] Master 127.0.97.62:40871 was elected leader, sending a full tablet report...
I20260927 22:47:05.743769 1437 ts_tablet_manager.cc:1428] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6: Time spent starting tablet: real 0.012s user 0.012s sys 0.002s
I20260927 22:47:05.747124 1436 ts_tablet_manager.cc:1428] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763: Time spent starting tablet: real 0.016s user 0.009s sys 0.009s
I20260927 22:47:05.749636 1438 raft_consensus.cc:348] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.749938 1438 raft_consensus.cc:374] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:05.750027 1438 raft_consensus.cc:729] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace44b01883147798952c214f6fe4e4f, State: Initialized, Role: FOLLOWER
I20260927 22:47:05.750485 1438 consensus_queue.cc:260] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.751391 1285 heartbeater.cc:503] Master 127.0.97.62:40871 was elected leader, sending a full tablet report...
I20260927 22:47:05.751933 1438 ts_tablet_manager.cc:1428] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 22:47:05.809088 1442 raft_consensus.cc:482] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:05.809360 1442 raft_consensus.cc:504] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.811208 1442 leader_election.cc:288] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06932bd982164400acddaef3bf870af6 (127.0.97.3:39183), ace44b01883147798952c214f6fe4e4f (127.0.97.2:42877)
I20260927 22:47:05.818789 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.811648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48643 (local address 127.0.97.3:39183)
0927 22:47:05.811850 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.811857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.812232 (+ 375us) server_negotiation.cc:407] Connection header received
0927 22:47:05.812410 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.812415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.812492 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.812588 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:05.813442 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.814429 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.816799 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.818029 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.818176 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.818309 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.818482 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.818585 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:47:05.819152 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.812548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36839 (local address 127.0.97.2:42877)
0927 22:47:05.812730 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.812735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.813398 (+ 663us) server_negotiation.cc:407] Connection header received
0927 22:47:05.813533 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.813539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.813607 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.813694 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:05.814074 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.814872 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.817055 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.818263 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.818385 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.818469 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.818809 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.818931 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:47:05.819227 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.811565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39183 (local address 127.0.97.1:48643)
0927 22:47:05.812116 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:05.812132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:05.812270 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:05.812655 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:05.812661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:05.812697 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:05.813284 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.813301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.814534 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.814538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.816628 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.816643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.818212 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.818216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.818326 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.818357 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:47:05.818455 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:47:05.818576 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":20,"thread_start_us":301,"threads_started":1}
I20260927 22:47:05.819198 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.812205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42877 (local address 127.0.97.1:36839)
0927 22:47:05.813302 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:05.813318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:05.813424 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:05.813697 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:05.813700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:05.813715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:05.813952 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.813960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.814993 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.814998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.816904 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:05.816914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.818488 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.818494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:05.818648 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.818679 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:47:05.818766 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 22:47:05.818867 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":57,"thread_start_us":775,"threads_started":1}
I20260927 22:47:05.820403 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b1cd836bd4028a38b216c4dd4d2ae" candidate_uuid: "5fd164a51da1481f99cf4c75a3624763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace44b01883147798952c214f6fe4e4f" is_pre_election: true
I20260927 22:47:05.820369 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b1cd836bd4028a38b216c4dd4d2ae" candidate_uuid: "5fd164a51da1481f99cf4c75a3624763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06932bd982164400acddaef3bf870af6" is_pre_election: true
I20260927 22:47:05.820919 1373 raft_consensus.cc:2454] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd164a51da1481f99cf4c75a3624763 in term 0.
I20260927 22:47:05.820919 1240 raft_consensus.cc:2454] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd164a51da1481f99cf4c75a3624763 in term 0.
I20260927 22:47:05.821573 1042 leader_election.cc:302] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fd164a51da1481f99cf4c75a3624763, ace44b01883147798952c214f6fe4e4f; no voters:
I20260927 22:47:05.821906 1442 raft_consensus.cc:2781] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:05.822048 1442 raft_consensus.cc:482] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:05.822135 1442 raft_consensus.cc:3037] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:05.824642 1442 raft_consensus.cc:504] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.825143 1442 leader_election.cc:288] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [CANDIDATE]: Term 1 election: Requested vote from peers 06932bd982164400acddaef3bf870af6 (127.0.97.3:39183), ace44b01883147798952c214f6fe4e4f (127.0.97.2:42877)
I20260927 22:47:05.825593 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b1cd836bd4028a38b216c4dd4d2ae" candidate_uuid: "5fd164a51da1481f99cf4c75a3624763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06932bd982164400acddaef3bf870af6"
I20260927 22:47:05.825595 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b1cd836bd4028a38b216c4dd4d2ae" candidate_uuid: "5fd164a51da1481f99cf4c75a3624763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace44b01883147798952c214f6fe4e4f"
I20260927 22:47:05.825795 1373 raft_consensus.cc:3037] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:05.825795 1240 raft_consensus.cc:3037] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:05.828853 1240 raft_consensus.cc:2454] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd164a51da1481f99cf4c75a3624763 in term 1.
I20260927 22:47:05.828850 1373 raft_consensus.cc:2454] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd164a51da1481f99cf4c75a3624763 in term 1.
I20260927 22:47:05.829208 1042 leader_election.cc:302] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06932bd982164400acddaef3bf870af6, 5fd164a51da1481f99cf4c75a3624763; no voters:
I20260927 22:47:05.829489 1442 raft_consensus.cc:2781] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:05.829879 1442 raft_consensus.cc:686] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [term 1 LEADER]: Becoming Leader. State: Replica: 5fd164a51da1481f99cf4c75a3624763, State: Running, Role: LEADER
I20260927 22:47:05.830309 1442 consensus_queue.cc:237] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } }
I20260927 22:47:05.834671 961 catalog_manager.cc:5495] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fd164a51da1481f99cf4c75a3624763 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5fd164a51da1481f99cf4c75a3624763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06932bd982164400acddaef3bf870af6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ace44b01883147798952c214f6fe4e4f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fd164a51da1481f99cf4c75a3624763" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34027 } health_report { overall_health: HEALTHY } } }
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:05.855854 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.842141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.0.97.1:34027)
0927 22:47:05.842277 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.842281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.843194 (+ 913us) server_negotiation.cc:407] Connection header received
0927 22:47:05.843374 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.843378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.843482 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.843559 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:05.844805 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.845507 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.846742 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.847158 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.847292 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.850840 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:05.850852 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:05.850868 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:05.850899 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:05.853535 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.854080 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.854087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.854090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.854256 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.854587 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.854592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.854594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.854953 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:05.855176 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.855341 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.855541 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 22:47:05.863241 1419 tablet.cc:2369] T df8b1cd836bd4028a38b216c4dd4d2ae P 06932bd982164400acddaef3bf870af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:05.897866 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.883767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.0.97.1:34027)
0927 22:47:05.883955 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.883962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.884415 (+ 453us) server_negotiation.cc:407] Connection header received
0927 22:47:05.884587 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.884593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.884669 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.884773 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:05.886170 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.886921 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.888292 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.888711 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.888838 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.892744 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:05.892750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:05.892754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:05.892791 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:05.895100 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.895777 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.895783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.895788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.895918 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.896349 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.896354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.896358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.896756 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:05.897036 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.897372 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.897556 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 22:47:05.916250 1286 tablet.cc:2369] T df8b1cd836bd4028a38b216c4dd4d2ae P ace44b01883147798952c214f6fe4e4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:47:05 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:05.938476 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.925147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.0.97.1:34027)
0927 22:47:05.925365 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.925372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.925578 (+ 206us) server_negotiation.cc:407] Connection header received
0927 22:47:05.925755 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.925759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.925838 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.925922 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:05.927205 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.928030 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.929244 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.929682 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.929790 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.933815 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:05.933820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:05.933823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:05.933856 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:05.936127 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.936733 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.936737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.936740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.936856 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.937176 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.937181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.937184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.937449 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:05.937685 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.937948 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.938167 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:47:05.940197 1066 tablet_service.cc:2908] Rejecting scan request for tablet df8b1cd836bd4028a38b216c4dd4d2ae: Uninitialized: clean time has not yet been initialized
I20260927 22:47:05.943555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1023
I20260927 22:47:05.955180 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:05.944372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.0.97.1:34027)
0927 22:47:05.944523 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:05.944529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:05.944808 (+ 279us) server_negotiation.cc:407] Connection header received
0927 22:47:05.944958 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:05.944964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:05.945040 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:05.945116 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:05.946204 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.946904 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.948111 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:05.948510 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:05.948612 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:05.950580 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:05.950585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:05.950589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:05.950616 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:05.952830 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.953428 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.953434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.953438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.953553 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:05.953891 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:05.953896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:05.953900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:05.954150 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:05.954374 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:05.954665 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:47:05.954887 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:47:05.955870 1146 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 22:47:06.043077 1152 tablet.cc:2369] T df8b1cd836bd4028a38b216c4dd4d2ae P 5fd164a51da1481f99cf4c75a3624763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:06.043710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 1023
I20260927 22:47:06.057173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1157
Sep 27 22:47:06 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:06.070464 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:06.057890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.0.97.2:42877)
0927 22:47:06.058066 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:06.058071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:06.058088 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:47:06.058233 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:06.058238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:06.058355 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:06.058439 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:06.059494 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:06.060209 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:06.061439 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:06.061845 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:06.061945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:06.065586 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:06.065599 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:06.065614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:06.065646 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:06.068122 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:06.068749 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:06.068757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:06.068759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:06.068894 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:06.069205 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:06.069208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:06.069210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:06.069609 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:06.069824 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:06.069995 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 22:47:06.070143 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:47:06.071079 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:06.155169 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 1157
I20260927 22:47:06.174068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1290
Sep 27 22:47:06 dist-test-slave-kr25 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:06.187808 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:06.174707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.0.97.3:39183)
0927 22:47:06.174864 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:06.174869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:06.174888 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:47:06.175034 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:06.175038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:06.175099 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:06.175183 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:06.176452 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:06.177151 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:06.178356 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:06.178728 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:06.178814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:06.182644 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:06.182658 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:06.182674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:06.182711 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:06.185307 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:06.185944 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:06.185952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:06.185954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:06.186109 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:06.186413 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:06.186416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:06.186418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:06.186758 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:06.186980 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:06.187250 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:47:06.187427 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:06.188373 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:06.274927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 1290
I20260927 22:47:06.287366 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 930
I20260927 22:47:06.288539 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:06.392212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 930
2026-09-27T22:47:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3772 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:06 dist-test-slave-kr25 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:06 dist-test-slave-kr25 krb5kdc[1466](info): set up 2 sockets
Sep 27 22:47:06 dist-test-slave-kr25 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260927 22:47:07.383345 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:07 dist-test-slave-kr25 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:07.547055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43611
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:07.847754 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:07.848021 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:07.848114 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:07.854985 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:07.855080 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:47:07.855118 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:07.855177 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:07.855232 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:07.855332 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:07.864895 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43611
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:07.866039 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:07.867249 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:07.878051 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:07.878024 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:07.878299 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:07.878741 1479 server_base.cc:1034] running on GCE node
I20260927 22:47:07.879467 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:07.880793 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:07.882030 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790549227881948 us; error 54 us; skew 500 ppm
I20260927 22:47:07.882356 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:07 dist-test-slave-kr25 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:07.890441 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:07.891826 1479 webserver.cc:460] Webserver started at http://127.0.97.62:45505/ using document root <none> and password file <none>
I20260927 22:47:07.892511 1479 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:07.892606 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:07.892905 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:07.895758 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "cd9cd3c3b98a40a7b77c06f1b828688a"
format_stamp: "Formatted at 2026-09-27 22:47:07 on dist-test-slave-kr25"
server_key: "4be7a965222ade6eaf527da1e0620f76"
server_key_iv: "f264f1dde5a84276e44107f62f1d0b30"
server_key_version: "encryptionkey@0"
I20260927 22:47:07.896495 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "cd9cd3c3b98a40a7b77c06f1b828688a"
format_stamp: "Formatted at 2026-09-27 22:47:07 on dist-test-slave-kr25"
server_key: "4be7a965222ade6eaf527da1e0620f76"
server_key_iv: "f264f1dde5a84276e44107f62f1d0b30"
server_key_version: "encryptionkey@0"
I20260927 22:47:07.901320 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:47:07.904614 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:07.905857 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:47:07.906003 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "cd9cd3c3b98a40a7b77c06f1b828688a"
format_stamp: "Formatted at 2026-09-27 22:47:07 on dist-test-slave-kr25"
server_key: "4be7a965222ade6eaf527da1e0620f76"
server_key_iv: "f264f1dde5a84276e44107f62f1d0b30"
server_key_version: "encryptionkey@0"
I20260927 22:47:07.906181 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:07.918565 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:07.922140 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:07.922369 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:07.941239 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43611
I20260927 22:47:07.941262 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43611 every 8 connection(s)
I20260927 22:47:07.942499 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:07.946033 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:07.951078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 1479
I20260927 22:47:07.951284 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:07.952082 388 external_mini_cluster.cc:1428] Setting key 61cd834f0800f4448578578bca48255c
I20260927 22:47:07.956756 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a: Bootstrap starting.
I20260927 22:47:07.959575 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:07.960490 1556 log.cc:824] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:07.963207 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a: No bootstrap required, opened a new log
Sep 27 22:47:07 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:07.972692 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43611 } }
I20260927 22:47:07.973156 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:07.973259 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9cd3c3b98a40a7b77c06f1b828688a, State: Initialized, Role: FOLLOWER
I20260927 22:47:07.973706 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:07.954070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.0.97.62:43611)
0927 22:47:07.954811 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:07.954825 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:07.954870 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:47:07.956488 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:07.956509 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:07.956878 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:07.957214 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:07.958481 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:07.959791 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:07.961013 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:07.961782 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:07.961959 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:07.965753 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:07.965767 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:07.965785 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:07.965826 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:07.969473 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:07.970108 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:07.970120 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:07.970129 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:07.970342 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:07.970636 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:07.970641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:07.970644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:07.971342 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:07.971660 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:07.971919 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 22:47:07.972336 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":331,"threads_started":1}
I20260927 22:47:07.973897 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43611 } }
I20260927 22:47:07.974176 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:07.974315 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:07.974491 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:07.976985 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43611 } }
I20260927 22:47:07.977437 1556 leader_election.cc:302] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9cd3c3b98a40a7b77c06f1b828688a; no voters:
I20260927 22:47:07.977941 1556 leader_election.cc:288] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:07.978053 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:07.978662 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [term 1 LEADER]: Becoming Leader. State: Replica: cd9cd3c3b98a40a7b77c06f1b828688a, State: Running, Role: LEADER
I20260927 22:47:07.979252 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43611 } }
I20260927 22:47:07.979996 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:07.981838 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9cd3c3b98a40a7b77c06f1b828688a. Latest consensus state: current_term: 1 leader_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43611 } } }
I20260927 22:47:07.982276 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:07.981870 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9cd3c3b98a40a7b77c06f1b828688a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43611 } } }
I20260927 22:47:07.983269 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:07.987478 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:07.992302 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:08.000324 1566 catalog_manager.cc:1347] Generated new cluster ID: b5b6df523a2544959da42c5df7e60fe0
I20260927 22:47:08.000437 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:08.011070 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:08.012358 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:08.018273 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a: Generated new TSK 0
I20260927 22:47:08.019284 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:08.062902 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43611
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:08.371171 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:08.371583 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:08.371737 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:08.379659 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:08.379762 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:08.379849 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:08.379928 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:08.389745 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:08.390986 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:08.392438 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:08.404721 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:08.404749 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:08.404796 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:08.405649 1582 server_base.cc:1034] running on GCE node
I20260927 22:47:08.406241 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:08.407110 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:08.408309 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790549228408264 us; error 59 us; skew 500 ppm
I20260927 22:47:08.408567 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:08 dist-test-slave-kr25 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:08.411667 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:08.413100 1582 webserver.cc:460] Webserver started at http://127.0.97.1:40439/ using document root <none> and password file <none>
I20260927 22:47:08.413791 1582 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:08.413897 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:08.414207 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:08.417057 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "f3b2f552e67a4edaa9a9c335a31aff2d"
format_stamp: "Formatted at 2026-09-27 22:47:08 on dist-test-slave-kr25"
server_key: "d46ad03b9510b9608769e6a1b78baeba"
server_key_iv: "6020f4f2a1630a4f7fdc2e9f7e2355eb"
server_key_version: "encryptionkey@0"
I20260927 22:47:08.417793 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f3b2f552e67a4edaa9a9c335a31aff2d"
format_stamp: "Formatted at 2026-09-27 22:47:08 on dist-test-slave-kr25"
server_key: "d46ad03b9510b9608769e6a1b78baeba"
server_key_iv: "6020f4f2a1630a4f7fdc2e9f7e2355eb"
server_key_version: "encryptionkey@0"
I20260927 22:47:08.422693 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:47:08.425980 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:08.427240 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:08.427381 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "f3b2f552e67a4edaa9a9c335a31aff2d"
format_stamp: "Formatted at 2026-09-27 22:47:08 on dist-test-slave-kr25"
server_key: "d46ad03b9510b9608769e6a1b78baeba"
server_key_iv: "6020f4f2a1630a4f7fdc2e9f7e2355eb"
server_key_version: "encryptionkey@0"
I20260927 22:47:08.427623 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:08.440356 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:08.444219 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:08.444478 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:08.447009 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:08.447099 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:08.447222 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:08.447286 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:47:08 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:08.470399 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.447821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49141 (local address 127.0.97.62:43611)
0927 22:47:08.448013 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:08.448019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:08.449387 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 22:47:08.452121 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:08.452128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:08.452219 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:08.452326 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:08.454448 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.455375 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.458181 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.458683 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.458807 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.463862 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:08.463868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:08.463873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:08.463915 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:08.467212 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.468034 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.468041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.468045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.468208 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.468621 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.468626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.468629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.469027 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:08.469174 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:08.469798 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 22:47:08.470026 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:08.471230 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.448372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43611 (local address 127.0.97.1:49141)
0927 22:47:08.449207 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:08.449255 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:08.451789 (+ 2534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:08.452657 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:08.452669 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:08.453416 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:08.454245 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.454278 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.455572 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.455578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.458005 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.458044 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.459015 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.459021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.459308 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.460128 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:08.460162 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:08.463560 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:08.467363 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.467372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.467387 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.467903 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.468319 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.468322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.468324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.468520 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.469146 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:08.469152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:08.469472 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:47:08.469687 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 22:47:08.470048 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":314,"threads_started":1}
I20260927 22:47:08.491174 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.477561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55175 (local address 127.0.97.62:43611)
0927 22:47:08.477762 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:08.477769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:08.478207 (+ 438us) server_negotiation.cc:407] Connection header received
0927 22:47:08.478387 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:08.478391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:08.478452 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:08.478529 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:08.479755 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.480462 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.482445 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.483010 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.483151 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.485509 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:08.485515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:08.485518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:08.485547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:08.488358 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.489224 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.489229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.489233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.489400 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.489793 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.489798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.489802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.490128 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:08.490250 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:08.490620 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 22:47:08.490829 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:47:08.491314 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.477480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43611 (local address 127.0.97.1:55175)
0927 22:47:08.478097 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:08.478120 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:08.478275 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:08.478578 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:08.478583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:08.479190 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:08.479618 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.479632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.480596 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.480601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.482271 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.482287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.483430 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.483474 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.483699 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.484450 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:08.484467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:08.485303 (+ 836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:08.488474 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.488483 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.488487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.489047 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.489530 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.489535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.489539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.489663 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.490235 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:08.490239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:08.490408 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 22:47:08.490523 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 22:47:08.490870 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":297,"threads_started":1}
I20260927 22:47:08.494005 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42371
I20260927 22:47:08.494014 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42371 every 8 connection(s)
I20260927 22:47:08.495258 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:08.497066 1717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43611
I20260927 22:47:08.497342 1717 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:08.497977 1717 heartbeater.cc:511] Master 127.0.97.62:43611 requested a full tablet report, sending...
I20260927 22:47:08.499655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 1582
I20260927 22:47:08.499830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:08.499840 1511 ts_manager.cc:194] Registered new tserver with Master: f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371)
I20260927 22:47:08.500404 388 external_mini_cluster.cc:1428] Setting key fe40fa11bf3a934aad43cc8b9da18490
I20260927 22:47:08.501379 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49141
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:08.536047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43611
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:08.841356 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:08.841620 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:08.841749 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:08.848723 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:08.848826 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:08.848948 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:08.849057 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:08.858881 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:08.860133 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:08.861400 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:08.873536 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:08.873617 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:08.873586 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:08.874436 1721 server_base.cc:1034] running on GCE node
I20260927 22:47:08.875033 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:08.875928 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:08.877130 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790549228877067 us; error 36 us; skew 500 ppm
I20260927 22:47:08.877408 1721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:08 dist-test-slave-kr25 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:08.880354 1721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:08.881687 1721 webserver.cc:460] Webserver started at http://127.0.97.2:45543/ using document root <none> and password file <none>
I20260927 22:47:08.882313 1721 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:08.882395 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:08.882658 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:08.885378 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "c61ea5fd47404af18eb8ef9a5067fd30"
format_stamp: "Formatted at 2026-09-27 22:47:08 on dist-test-slave-kr25"
server_key: "3132f25cf734a0791c74c31f323b1216"
server_key_iv: "886a3ad585ac4b761faf7a39a13ac25f"
server_key_version: "encryptionkey@0"
I20260927 22:47:08.886067 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c61ea5fd47404af18eb8ef9a5067fd30"
format_stamp: "Formatted at 2026-09-27 22:47:08 on dist-test-slave-kr25"
server_key: "3132f25cf734a0791c74c31f323b1216"
server_key_iv: "886a3ad585ac4b761faf7a39a13ac25f"
server_key_version: "encryptionkey@0"
I20260927 22:47:08.890833 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:47:08.894089 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:08.895349 1721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:08.895539 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "c61ea5fd47404af18eb8ef9a5067fd30"
format_stamp: "Formatted at 2026-09-27 22:47:08 on dist-test-slave-kr25"
server_key: "3132f25cf734a0791c74c31f323b1216"
server_key_iv: "886a3ad585ac4b761faf7a39a13ac25f"
server_key_version: "encryptionkey@0"
I20260927 22:47:08.895725 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:08.909552 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:08.913640 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:08.913880 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:08.916499 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:08.916584 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:08.916733 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:08.916796 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:47:08 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:08.937152 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.916867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38723 (local address 127.0.97.62:43611)
0927 22:47:08.917077 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:08.917083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:08.918437 (+ 1354us) server_negotiation.cc:407] Connection header received
0927 22:47:08.920584 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:08.920591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:08.920681 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:08.920797 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:08.922729 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.923738 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.926018 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.926534 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.926645 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.930727 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:08.930732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:08.930735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:08.930770 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:08.933607 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.934499 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.934505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.934508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.934669 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.935099 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.935104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.935108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.935414 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:08.935581 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:08.936569 (+ 988us) server_negotiation.cc:299] Negotiation successful
0927 22:47:08.936801 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:47:08.938926 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.917478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43611 (local address 127.0.97.2:38723)
0927 22:47:08.918272 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:08.918318 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:08.920242 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:08.921048 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:08.921056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:08.921758 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:08.922559 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.922585 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.923951 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.923956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.925824 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.925838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.926760 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.926767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.927005 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.927764 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:08.927785 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:08.930511 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:08.933742 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.933753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.933771 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.934365 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.934780 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.934783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.934785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.934989 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.935569 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:08.935579 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:08.936097 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 22:47:08.936477 (+ 380us) client_negotiation.cc:240] Negotiation successful
0927 22:47:08.937107 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":351,"threads_started":1}
I20260927 22:47:08.959969 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.944404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59645 (local address 127.0.97.62:43611)
0927 22:47:08.944575 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:08.944581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:08.947983 (+ 3402us) server_negotiation.cc:407] Connection header received
0927 22:47:08.948131 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:08.948136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:08.948211 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:08.948308 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:08.949228 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.950298 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.951630 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.952128 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.952240 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.954592 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:08.954598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:08.954603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:08.954639 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:08.957599 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.958176 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.958181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.958183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.958297 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:08.958621 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:08.958626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:08.958629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:08.958900 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:08.959026 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:08.959426 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 22:47:08.959656 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:47:08.960031 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:08.944406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43611 (local address 127.0.97.2:59645)
0927 22:47:08.947888 (+ 3482us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:08.947910 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:08.948028 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:08.948432 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:08.948436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:08.948839 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:08.949102 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.949110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.950424 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.950429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.951460 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:08.951475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:08.952296 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:08.952302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:08.952475 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:08.953178 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:08.953204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:08.954377 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:08.957707 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.957712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.957716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.958078 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.958397 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:08.958401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:08.958405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:08.958531 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:08.959012 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:08.959016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:08.959179 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 22:47:08.959297 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:47:08.959598 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3336,"thread_start_us":3202,"threads_started":1}
I20260927 22:47:08.968786 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32829
I20260927 22:47:08.968806 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32829 every 8 connection(s)
I20260927 22:47:08.969895 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:08.970834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 1721
I20260927 22:47:08.971024 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:08.971526 388 external_mini_cluster.cc:1428] Setting key 1b18d876dd1e8a53365ee9351811383c
I20260927 22:47:08.971894 1856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43611
I20260927 22:47:08.972121 1856 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:08.972715 1856 heartbeater.cc:511] Master 127.0.97.62:43611 requested a full tablet report, sending...
I20260927 22:47:08.973935 1511 ts_manager.cc:194] Registered new tserver with Master: c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2:32829)
I20260927 22:47:08.974697 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38723
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:09.007341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43611
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:09.309084 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:09.309407 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:09.309549 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:09.316906 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:09.317039 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:09.317153 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:09.317262 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:09.326978 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:09.328302 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:09.329572 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:09.341675 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:09.341691 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:09.341720 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:09.342716 1860 server_base.cc:1034] running on GCE node
I20260927 22:47:09.343268 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:09.344165 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:09.345376 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790549229345310 us; error 40 us; skew 500 ppm
I20260927 22:47:09.345643 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:09 dist-test-slave-kr25 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:09.348536 1860 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:09.349779 1860 webserver.cc:460] Webserver started at http://127.0.97.3:33071/ using document root <none> and password file <none>
I20260927 22:47:09.350450 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:09.350543 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:09.350858 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:09.353663 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "733cc2855c594792af21fa0069f44092"
format_stamp: "Formatted at 2026-09-27 22:47:09 on dist-test-slave-kr25"
server_key: "788108915b9b3354cbb7a150c8c8ecbb"
server_key_iv: "74125879ace53b04680c2269346c55ed"
server_key_version: "encryptionkey@0"
I20260927 22:47:09.354393 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "733cc2855c594792af21fa0069f44092"
format_stamp: "Formatted at 2026-09-27 22:47:09 on dist-test-slave-kr25"
server_key: "788108915b9b3354cbb7a150c8c8ecbb"
server_key_iv: "74125879ace53b04680c2269346c55ed"
server_key_version: "encryptionkey@0"
I20260927 22:47:09.359066 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:47:09.362267 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:09.363525 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:09.363662 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "733cc2855c594792af21fa0069f44092"
format_stamp: "Formatted at 2026-09-27 22:47:09 on dist-test-slave-kr25"
server_key: "788108915b9b3354cbb7a150c8c8ecbb"
server_key_iv: "74125879ace53b04680c2269346c55ed"
server_key_version: "encryptionkey@0"
I20260927 22:47:09.363873 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:09.389492 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:09.393680 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:09.393951 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:09.396632 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:09.396723 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:09.396863 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:09.396965 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 22:47:09 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:09.419355 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.397116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42017 (local address 127.0.97.62:43611)
0927 22:47:09.397308 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.397315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.398710 (+ 1395us) server_negotiation.cc:407] Connection header received
0927 22:47:09.400982 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.400988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.401084 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.401192 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:09.403937 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.404839 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.407640 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.408188 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.408315 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.412749 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:09.412756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:09.412761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:09.412803 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:09.415967 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.416667 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.416673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.416676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.416827 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.417366 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.417372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.417375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.417724 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:09.417866 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.418798 (+ 932us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.419027 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:47:09.420396 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.397720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43611 (local address 127.0.97.3:42017)
0927 22:47:09.398550 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.398596 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.400601 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.401639 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.401669 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.402632 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:09.403664 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.403692 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.405022 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.405028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.407484 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.407502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.408517 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.408523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.408823 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.409618 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:09.409640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:09.412464 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:09.416111 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:09.416121 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:09.416139 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:09.416542 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:09.416960 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:09.416965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:09.416969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:09.417221 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:09.417882 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:09.417891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:09.418333 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.418693 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.419238 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":350,"threads_started":1}
I20260927 22:47:09.439534 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.425272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35509 (local address 127.0.97.62:43611)
0927 22:47:09.425446 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.425452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.426571 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 22:47:09.426742 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.426746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.426803 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.426881 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:09.427922 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.428857 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.430772 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.431332 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.431472 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.433753 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:09.433757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:09.433761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:09.433789 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:09.436826 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.437573 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.437578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.437582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.437757 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.438138 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.438144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.438147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.438411 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:09.438559 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.438941 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.439131 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:47:09.439633 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.425316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43611 (local address 127.0.97.3:35509)
0927 22:47:09.426465 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.426486 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.426626 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.426939 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.426943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.427377 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:09.427737 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.427748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.429022 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.429026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.430592 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.430606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.431728 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.431733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.431923 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.432517 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:09.432535 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:09.433530 (+ 995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:09.436954 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:09.436959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:09.436963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:09.437438 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:09.437874 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:09.437878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:09.437882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:09.438036 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:09.438524 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:09.438528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:09.438697 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.438822 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.439141 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":362,"threads_started":1}
I20260927 22:47:09.441913 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36395
I20260927 22:47:09.441936 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36395 every 8 connection(s)
I20260927 22:47:09.443289 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:09.445092 1995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43611
I20260927 22:47:09.445274 1995 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:09.445822 1995 heartbeater.cc:511] Master 127.0.97.62:43611 requested a full tablet report, sending...
I20260927 22:47:09.446964 1511 ts_manager.cc:194] Registered new tserver with Master: 733cc2855c594792af21fa0069f44092 (127.0.97.3:36395)
I20260927 22:47:09.447768 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42017
I20260927 22:47:09.452102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 1860
I20260927 22:47:09.452255 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:09.452668 388 external_mini_cluster.cc:1428] Setting key 52ab22bb71b1197ee19d8b7ae2e2c691
I20260927 22:47:09.455224 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:47:09 dist-test-slave-kr25 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:47:09 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:09.496179 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.482988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.0.97.62:43611)
0927 22:47:09.483199 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.483206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.483358 (+ 152us) server_negotiation.cc:407] Connection header received
0927 22:47:09.483555 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.483559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.483639 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.483724 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:09.484952 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.485645 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.486800 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.487201 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.487289 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.490778 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:09.490784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:09.490787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:09.490821 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:09.493836 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.494370 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.494376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.494379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.494533 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.494837 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.494842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.494846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.495177 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:09.495423 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.495661 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.495876 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:47:09.503831 1717 heartbeater.cc:503] Master 127.0.97.62:43611 was elected leader, sending a full tablet report...
I20260927 22:47:09.500273 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:09.506860 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:09.529145 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.518023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42371 (local address 127.0.0.1:47906)
0927 22:47:09.521128 (+ 3105us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.521143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.521421 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.522376 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.522380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.522403 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:09.522714 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.522726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.524349 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.524354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.526627 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.526640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.528356 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.528362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.528536 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.528558 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.528634 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.528749 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2979,"mutex_wait_us":3,"thread_start_us":3593,"threads_started":1}
I20260927 22:47:09.529237 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.517674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.2:32829)
0927 22:47:09.520291 (+ 2617us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.520297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.520701 (+ 404us) server_negotiation.cc:407] Connection header received
0927 22:47:09.521034 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.521043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.521297 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.521473 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:09.522387 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.523946 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.526545 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.527828 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.527951 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.528140 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.528843 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.528985 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2508,"thread_start_us":2423,"threads_started":1}
I20260927 22:47:09.529304 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.518892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.0.97.3:36395)
0927 22:47:09.519715 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.519725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.521965 (+ 2240us) server_negotiation.cc:407] Connection header received
0927 22:47:09.522224 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.522231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.522453 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.522595 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:09.523338 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.524621 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.526960 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.527904 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.528018 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.528292 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.528794 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.528950 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":483,"threads_started":1}
I20260927 22:47:09.529632 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.517584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32829 (local address 127.0.0.1:57636)
0927 22:47:09.520570 (+ 2986us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.520597 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.520757 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.521751 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.521758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.521797 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:09.522231 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.522243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.524081 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.524086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.526401 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.526412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.528029 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.528035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.528210 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.528258 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.528408 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.528586 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2821,"mutex_wait_us":194,"thread_start_us":420,"threads_started":1}
I20260927 22:47:09.530159 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.518085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36395 (local address 127.0.0.1:56012)
0927 22:47:09.521862 (+ 3777us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.521879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.522051 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.522780 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.522785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.522808 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:09.523133 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.523142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.524752 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.524757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.526834 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.526843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.528423 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.528428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.528592 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.528620 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.529473 (+ 853us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.529600 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3668,"mutex_wait_us":46,"thread_start_us":2800,"threads_started":1}
I20260927 22:47:09.530256 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.518497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.0.97.1:42371)
0927 22:47:09.520078 (+ 1581us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.520086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.521248 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 22:47:09.521983 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.521991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.522201 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.522381 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:09.522888 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.524230 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.526772 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.528168 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.528276 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.528443 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.528694 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.530001 (+ 1307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1460,"thread_start_us":1336,"threads_started":1}
I20260927 22:47:09.533288 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa1c591c85f46d889cc0e2b642eb95b (DEFAULT_TABLE table=test-table [id=63140365a75941d3a62ac966b4fac303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:09.533406 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa1c591c85f46d889cc0e2b642eb95b (DEFAULT_TABLE table=test-table [id=63140365a75941d3a62ac966b4fac303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:09.533406 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa1c591c85f46d889cc0e2b642eb95b (DEFAULT_TABLE table=test-table [id=63140365a75941d3a62ac966b4fac303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:09.534544 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa1c591c85f46d889cc0e2b642eb95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.534621 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa1c591c85f46d889cc0e2b642eb95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.534898 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa1c591c85f46d889cc0e2b642eb95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.544795 2014 tablet_bootstrap.cc:492] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Bootstrap starting.
I20260927 22:47:09.547962 2014 tablet_bootstrap.cc:654] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.548148 2015 tablet_bootstrap.cc:492] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Bootstrap starting.
I20260927 22:47:09.549197 2014 log.cc:824] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:09.550380 2016 tablet_bootstrap.cc:492] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Bootstrap starting.
I20260927 22:47:09.551411 2015 tablet_bootstrap.cc:654] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.551954 2014 tablet_bootstrap.cc:492] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: No bootstrap required, opened a new log
I20260927 22:47:09.552186 2014 ts_tablet_manager.cc:1397] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:47:09.552690 2015 log.cc:824] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:09.553547 2016 tablet_bootstrap.cc:654] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.554600 2016 log.cc:824] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:09.555549 2015 tablet_bootstrap.cc:492] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: No bootstrap required, opened a new log
I20260927 22:47:09.555835 2015 ts_tablet_manager.cc:1397] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 22:47:09.557497 2016 tablet_bootstrap.cc:492] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: No bootstrap required, opened a new log
I20260927 22:47:09.557724 2016 ts_tablet_manager.cc:1397] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:47:09.559923 2014 raft_consensus.cc:348] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.560233 2014 raft_consensus.cc:374] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.560355 2014 raft_consensus.cc:729] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b2f552e67a4edaa9a9c335a31aff2d, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.560896 2014 consensus_queue.cc:260] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.562239 2014 ts_tablet_manager.cc:1428] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 22:47:09.567909 2015 raft_consensus.cc:348] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.568342 2015 raft_consensus.cc:374] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.568430 2015 raft_consensus.cc:729] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733cc2855c594792af21fa0069f44092, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.568998 2015 consensus_queue.cc:260] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.570032 1995 heartbeater.cc:503] Master 127.0.97.62:43611 was elected leader, sending a full tablet report...
I20260927 22:47:09.570016 2016 raft_consensus.cc:348] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.570415 2016 raft_consensus.cc:374] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.570518 2016 raft_consensus.cc:729] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61ea5fd47404af18eb8ef9a5067fd30, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.570907 2015 ts_tablet_manager.cc:1428] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 22:47:09.571130 2016 consensus_queue.cc:260] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.572150 1856 heartbeater.cc:503] Master 127.0.97.62:43611 was elected leader, sending a full tablet report...
I20260927 22:47:09.572613 2016 ts_tablet_manager.cc:1428] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 22:47:09.585202 2021 raft_consensus.cc:482] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:09.585453 2021 raft_consensus.cc:504] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.586923 2021 leader_election.cc:288] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2:32829), f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371)
I20260927 22:47:09.593339 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.587276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54803 (local address 127.0.97.1:42371)
0927 22:47:09.587476 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.587482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.587581 (+ 99us) server_negotiation.cc:407] Connection header received
0927 22:47:09.587815 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.587820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.587909 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.588022 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:09.588428 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.589336 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.591405 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.592570 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.592664 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.592736 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.593079 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.593166 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:47:09.593497 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.587190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42371 (local address 127.0.97.3:54803)
0927 22:47:09.587487 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.587507 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.587650 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.588032 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.588037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.588064 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:09.588278 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.588290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.589456 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.589462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.591256 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.591267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.592722 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.592728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.592917 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.592952 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.593060 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.593204 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":20,"spinlock_wait_cycles":12672}
I20260927 22:47:09.594409 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.587261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32829 (local address 127.0.97.3:53159)
0927 22:47:09.588016 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.588032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.588155 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.588493 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.588498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.588521 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:09.588900 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.588911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.590190 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.590193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.592333 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.592347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.593769 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.593774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.593923 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.593954 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.594042 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.594149 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":1,"thread_start_us":393,"threads_started":1}
I20260927 22:47:09.594451 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.587276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53159 (local address 127.0.97.2:32829)
0927 22:47:09.587476 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.587481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.588103 (+ 622us) server_negotiation.cc:407] Connection header received
0927 22:47:09.588270 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.588315 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.588392 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.588458 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:09.589108 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.590071 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.592483 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.593551 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.593678 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.593770 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.594069 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.594184 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:47:09.594591 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa1c591c85f46d889cc0e2b642eb95b" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" is_pre_election: true
I20260927 22:47:09.595152 1667 raft_consensus.cc:2454] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 0.
I20260927 22:47:09.595321 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa1c591c85f46d889cc0e2b642eb95b" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" is_pre_election: true
I20260927 22:47:09.595809 1877 leader_election.cc:302] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733cc2855c594792af21fa0069f44092, f3b2f552e67a4edaa9a9c335a31aff2d; no voters:
I20260927 22:47:09.595817 1806 raft_consensus.cc:2454] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 0.
I20260927 22:47:09.596141 2021 raft_consensus.cc:2781] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:09.596289 2021 raft_consensus.cc:482] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:09.596405 2021 raft_consensus.cc:3037] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:09.598482 2021 raft_consensus.cc:504] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.599030 2021 leader_election.cc:288] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 election: Requested vote from peers c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2:32829), f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371)
I20260927 22:47:09.599332 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa1c591c85f46d889cc0e2b642eb95b" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61ea5fd47404af18eb8ef9a5067fd30"
I20260927 22:47:09.599525 1806 raft_consensus.cc:3037] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:09.599515 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa1c591c85f46d889cc0e2b642eb95b" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d"
I20260927 22:47:09.599684 1667 raft_consensus.cc:3037] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:09.601792 1806 raft_consensus.cc:2454] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 1.
I20260927 22:47:09.601854 1667 raft_consensus.cc:2454] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 1.
I20260927 22:47:09.602144 1876 leader_election.cc:302] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733cc2855c594792af21fa0069f44092, c61ea5fd47404af18eb8ef9a5067fd30; no voters:
I20260927 22:47:09.602415 2021 raft_consensus.cc:2781] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:09.603199 2021 raft_consensus.cc:686] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 1 LEADER]: Becoming Leader. State: Replica: 733cc2855c594792af21fa0069f44092, State: Running, Role: LEADER
I20260927 22:47:09.603664 2021 consensus_queue.cc:237] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.608040 1510 catalog_manager.cc:5495] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 733cc2855c594792af21fa0069f44092 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "733cc2855c594792af21fa0069f44092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } health_report { overall_health: HEALTHY } } }
I20260927 22:47:09.631033 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.626126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.0.97.3:36395)
0927 22:47:09.626310 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.626316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.626456 (+ 140us) server_negotiation.cc:407] Connection header received
0927 22:47:09.626709 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.626713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.626801 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.626909 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:09.627155 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.627885 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.629090 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.629511 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.629597 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.630005 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:09.630510 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:09.630615 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.630719 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.630790 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:47:09.638582 1667 raft_consensus.cc:1261] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Refusing update from remote peer 733cc2855c594792af21fa0069f44092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:09.638582 1806 raft_consensus.cc:1261] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Refusing update from remote peer 733cc2855c594792af21fa0069f44092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:09.639914 2024 consensus_queue.cc:1035] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 22:47:09.640239 2021 consensus_queue.cc:1035] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:09.651099 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334089644590788608 to 7334089644460032000. Current Snapshot: MvccSnapshot[applied={T|T < 7334089644590788608}]
I20260927 22:47:09.668476 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:09.668709 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:09.672078 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a: Sending DeleteTablet for 3 replicas of tablet 0aa1c591c85f46d889cc0e2b642eb95b
I20260927 22:47:09.673168 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa1c591c85f46d889cc0e2b642eb95b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47906
I20260927 22:47:09.673151 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa1c591c85f46d889cc0e2b642eb95b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57636
I20260927 22:47:09.673300 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa1c591c85f46d889cc0e2b642eb95b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56012
I20260927 22:47:09.675261 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:09.676427 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:09.684129 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 2abd0969eb684e238e1a665ce8f97899 (DEFAULT_TABLE table=test-table [id=29a60bef9b9a4fed9c5bbbe4988235ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:09.684716 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abd0969eb684e238e1a665ce8f97899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.685060 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 2abd0969eb684e238e1a665ce8f97899 (DEFAULT_TABLE table=test-table [id=29a60bef9b9a4fed9c5bbbe4988235ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:09.685623 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abd0969eb684e238e1a665ce8f97899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.688143 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 2abd0969eb684e238e1a665ce8f97899 (DEFAULT_TABLE table=test-table [id=29a60bef9b9a4fed9c5bbbe4988235ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:09.688692 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abd0969eb684e238e1a665ce8f97899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.691514 2015 tablet_bootstrap.cc:492] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Bootstrap starting.
I20260927 22:47:09.693142 2015 tablet_bootstrap.cc:654] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.697268 2039 tablet_replica.cc:330] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: stopping tablet replica
I20260927 22:47:09.697691 2039 raft_consensus.cc:2229] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:09.699635 2039 raft_consensus.cc:2258] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:09.700718 2016 tablet_bootstrap.cc:492] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Bootstrap starting.
I20260927 22:47:09.701893 2014 tablet_bootstrap.cc:492] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Bootstrap starting.
I20260927 22:47:09.702221 2016 tablet_bootstrap.cc:654] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.703750 2014 tablet_bootstrap.cc:654] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.704625 2038 tablet_replica.cc:330] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: stopping tablet replica
I20260927 22:47:09.704890 2016 tablet_bootstrap.cc:492] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: No bootstrap required, opened a new log
I20260927 22:47:09.705595 2038 raft_consensus.cc:2229] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:09.706212 2016 ts_tablet_manager.cc:1397] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 22:47:09.706565 2037 tablet_replica.cc:330] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: stopping tablet replica
I20260927 22:47:09.707131 2038 raft_consensus.cc:2258] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:09.707129 2016 raft_consensus.cc:348] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.707468 2016 raft_consensus.cc:374] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.707605 2016 raft_consensus.cc:729] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61ea5fd47404af18eb8ef9a5067fd30, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.707979 2037 raft_consensus.cc:2229] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:09.708492 2037 raft_consensus.cc:2258] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:09.709497 2016 consensus_queue.cc:260] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.710619 2039 ts_tablet_manager.cc:1905] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:09.710964 2015 tablet_bootstrap.cc:492] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: No bootstrap required, opened a new log
I20260927 22:47:09.711158 2015 ts_tablet_manager.cc:1397] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.003s
I20260927 22:47:09.711205 2016 ts_tablet_manager.cc:1428] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260927 22:47:09.712095 2015 raft_consensus.cc:348] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.712378 2015 raft_consensus.cc:374] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.712536 2015 raft_consensus.cc:729] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733cc2855c594792af21fa0069f44092, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.712867 2015 consensus_queue.cc:260] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.714066 2037 ts_tablet_manager.cc:1905] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:09.715804 2015 ts_tablet_manager.cc:1428] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 22:47:09.718173 2038 ts_tablet_manager.cc:1905] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:09.721292 2014 tablet_bootstrap.cc:492] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: No bootstrap required, opened a new log
W20260927 22:47:09.721673 1857 tablet.cc:2369] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:09.722121 2039 ts_tablet_manager.cc:1918] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:09.722270 2039 log.cc:1196] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/0aa1c591c85f46d889cc0e2b642eb95b
I20260927 22:47:09.722579 2014 ts_tablet_manager.cc:1397] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.004s
I20260927 22:47:09.722775 2039 ts_tablet_manager.cc:1939] T 0aa1c591c85f46d889cc0e2b642eb95b P 733cc2855c594792af21fa0069f44092: Deleting consensus metadata
I20260927 22:47:09.723544 2014 raft_consensus.cc:348] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.723834 2014 raft_consensus.cc:374] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.723951 2014 raft_consensus.cc:729] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b2f552e67a4edaa9a9c335a31aff2d, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.724251 2014 consensus_queue.cc:260] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.724447 1497 catalog_manager.cc:4841] TS 733cc2855c594792af21fa0069f44092 (127.0.97.3:36395): tablet 0aa1c591c85f46d889cc0e2b642eb95b (table test-table [id=63140365a75941d3a62ac966b4fac303]) successfully deleted
I20260927 22:47:09.725334 2014 ts_tablet_manager.cc:1428] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 22:47:09.726115 2037 ts_tablet_manager.cc:1918] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:09.726258 2037 log.cc:1196] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/0aa1c591c85f46d889cc0e2b642eb95b
I20260927 22:47:09.726759 2037 ts_tablet_manager.cc:1939] T 0aa1c591c85f46d889cc0e2b642eb95b P c61ea5fd47404af18eb8ef9a5067fd30: Deleting consensus metadata
I20260927 22:47:09.728461 1495 catalog_manager.cc:4841] TS c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2:32829): tablet 0aa1c591c85f46d889cc0e2b642eb95b (table test-table [id=63140365a75941d3a62ac966b4fac303]) successfully deleted
I20260927 22:47:09.730557 2038 ts_tablet_manager.cc:1918] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:09.730693 2038 log.cc:1196] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/0aa1c591c85f46d889cc0e2b642eb95b
I20260927 22:47:09.731088 2038 ts_tablet_manager.cc:1939] T 0aa1c591c85f46d889cc0e2b642eb95b P f3b2f552e67a4edaa9a9c335a31aff2d: Deleting consensus metadata
I20260927 22:47:09.732374 1496 catalog_manager.cc:4841] TS f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371): tablet 0aa1c591c85f46d889cc0e2b642eb95b (table test-table [id=63140365a75941d3a62ac966b4fac303]) successfully deleted
W20260927 22:47:09.747311 1718 tablet.cc:2369] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:09.846298 2032 raft_consensus.cc:482] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:09.846508 2032 raft_consensus.cc:504] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.847304 2032 leader_election.cc:288] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371), c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2:32829)
I20260927 22:47:09.847849 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abd0969eb684e238e1a665ce8f97899" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" is_pre_election: true
I20260927 22:47:09.847858 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abd0969eb684e238e1a665ce8f97899" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" is_pre_election: true
I20260927 22:47:09.848080 1667 raft_consensus.cc:2454] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 0.
I20260927 22:47:09.848100 1806 raft_consensus.cc:2454] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 0.
I20260927 22:47:09.848492 1877 leader_election.cc:302] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733cc2855c594792af21fa0069f44092, f3b2f552e67a4edaa9a9c335a31aff2d; no voters:
I20260927 22:47:09.848771 2032 raft_consensus.cc:2781] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:09.848904 2032 raft_consensus.cc:482] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:09.848990 2032 raft_consensus.cc:3037] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:09.851142 2032 raft_consensus.cc:504] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.851758 2032 leader_election.cc:288] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 election: Requested vote from peers f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371), c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2:32829)
I20260927 22:47:09.852243 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abd0969eb684e238e1a665ce8f97899" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d"
I20260927 22:47:09.852320 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abd0969eb684e238e1a665ce8f97899" candidate_uuid: "733cc2855c594792af21fa0069f44092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61ea5fd47404af18eb8ef9a5067fd30"
I20260927 22:47:09.852425 1667 raft_consensus.cc:3037] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:09.852528 1806 raft_consensus.cc:3037] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:09.855063 1667 raft_consensus.cc:2454] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 1.
I20260927 22:47:09.855334 1806 raft_consensus.cc:2454] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733cc2855c594792af21fa0069f44092 in term 1.
I20260927 22:47:09.855760 1877 leader_election.cc:302] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733cc2855c594792af21fa0069f44092, f3b2f552e67a4edaa9a9c335a31aff2d; no voters:
I20260927 22:47:09.856032 2032 raft_consensus.cc:2781] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:09.856176 2032 raft_consensus.cc:686] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 1 LEADER]: Becoming Leader. State: Replica: 733cc2855c594792af21fa0069f44092, State: Running, Role: LEADER
I20260927 22:47:09.856452 2032 consensus_queue.cc:237] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.859078 1510 catalog_manager.cc:5495] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 733cc2855c594792af21fa0069f44092 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "733cc2855c594792af21fa0069f44092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } health_report { overall_health: HEALTHY } } }
Sep 27 22:47:09 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:09.945255 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.929789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43611 (local address 127.0.0.1:58152)
0927 22:47:09.930429 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:09.930445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:09.930568 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:09.930915 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:09.930918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:09.931624 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:09.931915 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.931925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.933301 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.933304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.934631 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:09.934642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.935665 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.935669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:09.935800 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.936449 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:09.936465 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:09.938709 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:09.942102 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:09.942115 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:09.942124 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:09.943049 (+ 925us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:09.943403 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:09.943407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:09.943411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:09.943606 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:09.944202 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:09.944211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:09.944390 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 22:47:09.944580 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 22:47:09.944848 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":354,"threads_started":1}
I20260927 22:47:09.945379 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:09.929735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.0.97.62:43611)
0927 22:47:09.929959 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:09.929966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:09.930541 (+ 575us) server_negotiation.cc:407] Connection header received
0927 22:47:09.930681 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:09.930686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:09.930770 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:09.930897 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:09.932051 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.933180 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.934767 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:09.935505 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:09.935629 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:09.938889 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:09.938894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:09.938897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:09.938925 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:09.941956 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.943158 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.943163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.943167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.943289 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:09.943735 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:09.943740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:09.943743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:09.944070 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:09.944209 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:09.944702 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 22:47:09.945019 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":66}
W20260927 22:47:09.945832 1996 tablet.cc:2369] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:09.950704 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58152:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:47:09.951985 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:09.959543 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 80667ef84ed14590ad1ef0a4a9b8fe15 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c9ef3880547e4e078d8affc1f08f5b0f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:09.960108 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80667ef84ed14590ad1ef0a4a9b8fe15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.959903 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 80667ef84ed14590ad1ef0a4a9b8fe15 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c9ef3880547e4e078d8affc1f08f5b0f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:09.960440 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80667ef84ed14590ad1ef0a4a9b8fe15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.960251 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 80667ef84ed14590ad1ef0a4a9b8fe15 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c9ef3880547e4e078d8affc1f08f5b0f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:09.960786 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80667ef84ed14590ad1ef0a4a9b8fe15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:09.967129 2014 tablet_bootstrap.cc:492] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: Bootstrap starting.
I20260927 22:47:09.967397 2016 tablet_bootstrap.cc:492] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: Bootstrap starting.
I20260927 22:47:09.968152 2015 tablet_bootstrap.cc:492] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: Bootstrap starting.
I20260927 22:47:09.968904 2014 tablet_bootstrap.cc:654] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.969471 2016 tablet_bootstrap.cc:654] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.969815 2015 tablet_bootstrap.cc:654] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:09.978820 2014 tablet_bootstrap.cc:492] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: No bootstrap required, opened a new log
I20260927 22:47:09.979070 2014 ts_tablet_manager.cc:1397] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260927 22:47:09.980020 2014 raft_consensus.cc:348] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.980370 2014 raft_consensus.cc:374] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.980484 2014 raft_consensus.cc:729] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b2f552e67a4edaa9a9c335a31aff2d, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.980872 2014 consensus_queue.cc:260] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.981436 2020 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:09.981794 2020 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:09.983259 2014 ts_tablet_manager.cc:1428] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 22:47:09.984261 2015 tablet_bootstrap.cc:492] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: No bootstrap required, opened a new log
I20260927 22:47:09.984429 2015 ts_tablet_manager.cc:1397] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.006s
I20260927 22:47:09.985689 2016 tablet_bootstrap.cc:492] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: No bootstrap required, opened a new log
I20260927 22:47:09.985661 2015 raft_consensus.cc:348] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.985837 2016 ts_tablet_manager.cc:1397] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20260927 22:47:09.985882 2015 raft_consensus.cc:374] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.985946 2015 raft_consensus.cc:729] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733cc2855c594792af21fa0069f44092, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.986176 2015 consensus_queue.cc:260] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.986532 2016 raft_consensus.cc:348] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.986757 2016 raft_consensus.cc:374] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:09.986768 2015 ts_tablet_manager.cc:1428] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:47:09.986826 2016 raft_consensus.cc:729] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61ea5fd47404af18eb8ef9a5067fd30, State: Initialized, Role: FOLLOWER
I20260927 22:47:09.987053 2016 consensus_queue.cc:260] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:09.987465 2022 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:09.986616 2032 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:09.987807 2022 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:09.987840 2032 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:09.988181 2016 ts_tablet_manager.cc:1428] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:47:10.325374 2022 raft_consensus.cc:482] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:10.325574 2022 raft_consensus.cc:504] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:10.326220 2032 consensus_queue.cc:1035] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:10.327236 2022 leader_election.cc:288] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371), 733cc2855c594792af21fa0069f44092 (127.0.97.3:36395)
I20260927 22:47:10.342279 2021 consensus_queue.cc:1035] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:10.346553 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.327681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36395 (local address 127.0.97.2:49869)
0927 22:47:10.337910 (+ 10229us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:10.337930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:10.338114 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:10.338606 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:10.338610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:10.338637 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:10.338863 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.338874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.341516 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.341520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.343935 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.343948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.345597 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.345605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.345782 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.345819 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:47:10.345910 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:47:10.346042 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10104,"mutex_wait_us":532,"thread_start_us":9662,"threads_started":1}
I20260927 22:47:10.346606 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.328066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49869 (local address 127.0.97.3:36395)
0927 22:47:10.337500 (+ 9434us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.337506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.338025 (+ 519us) server_negotiation.cc:407] Connection header received
0927 22:47:10.338263 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.338270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.338339 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.338404 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:10.339036 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.341310 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.344108 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.345268 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.345405 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.345506 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.346190 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.346326 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9320,"thread_start_us":9215,"threads_started":1}
I20260927 22:47:10.346917 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.327392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42371 (local address 127.0.97.2:50453)
0927 22:47:10.337499 (+ 10107us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:10.337521 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:10.337683 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:10.339792 (+ 2109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:10.339797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:10.339821 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:10.340160 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.340172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.341577 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.341582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.344619 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.344631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.346189 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.346195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.346376 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.346410 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:47:10.346506 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:47:10.346631 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9978,"thread_start_us":10371,"threads_started":1}
I20260927 22:47:10.347570 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.327414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50453 (local address 127.0.97.1:42371)
0927 22:47:10.339352 (+ 11938us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.339357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.339389 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:47:10.339467 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.339471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.339532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.339624 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:10.340335 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.341432 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.344761 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.345900 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.346020 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.346112 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.347136 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.347269 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11837,"thread_start_us":11723,"threads_started":1}
I20260927 22:47:10.348008 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80667ef84ed14590ad1ef0a4a9b8fe15" candidate_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" is_pre_election: true
I20260927 22:47:10.348248 1667 raft_consensus.cc:2454] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c61ea5fd47404af18eb8ef9a5067fd30 in term 0.
I20260927 22:47:10.348214 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80667ef84ed14590ad1ef0a4a9b8fe15" candidate_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733cc2855c594792af21fa0069f44092" is_pre_election: true
I20260927 22:47:10.348546 1945 raft_consensus.cc:2454] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c61ea5fd47404af18eb8ef9a5067fd30 in term 0.
I20260927 22:47:10.348799 1738 leader_election.cc:302] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c61ea5fd47404af18eb8ef9a5067fd30, f3b2f552e67a4edaa9a9c335a31aff2d; no voters:
I20260927 22:47:10.349211 2022 raft_consensus.cc:2781] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:10.349336 2022 raft_consensus.cc:482] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:10.349445 2022 raft_consensus.cc:3037] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:10.351874 2022 raft_consensus.cc:504] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:10.352308 2022 leader_election.cc:288] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [CANDIDATE]: Term 1 election: Requested vote from peers f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371), 733cc2855c594792af21fa0069f44092 (127.0.97.3:36395)
I20260927 22:47:10.352720 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80667ef84ed14590ad1ef0a4a9b8fe15" candidate_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733cc2855c594792af21fa0069f44092"
I20260927 22:47:10.352734 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80667ef84ed14590ad1ef0a4a9b8fe15" candidate_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d"
I20260927 22:47:10.352917 1667 raft_consensus.cc:3037] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:10.352917 1945 raft_consensus.cc:3037] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:10.355670 1667 raft_consensus.cc:2454] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c61ea5fd47404af18eb8ef9a5067fd30 in term 1.
I20260927 22:47:10.355685 1945 raft_consensus.cc:2454] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c61ea5fd47404af18eb8ef9a5067fd30 in term 1.
I20260927 22:47:10.356042 1739 leader_election.cc:302] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 733cc2855c594792af21fa0069f44092, c61ea5fd47404af18eb8ef9a5067fd30, f3b2f552e67a4edaa9a9c335a31aff2d; no voters:
I20260927 22:47:10.356321 2022 raft_consensus.cc:2781] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:10.356660 2022 raft_consensus.cc:686] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 LEADER]: Becoming Leader. State: Replica: c61ea5fd47404af18eb8ef9a5067fd30, State: Running, Role: LEADER
I20260927 22:47:10.356973 2022 consensus_queue.cc:237] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } }
I20260927 22:47:10.358538 2062 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: TxnStatusTablet state changed. Reason: New leader c61ea5fd47404af18eb8ef9a5067fd30. Latest consensus state: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:10.358804 2062 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:10.359467 2064 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:47:10.359651 2064 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:47:10.360002 1511 catalog_manager.cc:5495] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 reported cstate change: term changed from 0 to 1, leader changed from <none> to c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:10.425563 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.419622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32829 (local address 127.0.0.1:57638)
0927 22:47:10.419822 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:10.419837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:10.419935 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:10.420798 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:10.420803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:10.420816 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:10.420998 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.421009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.422184 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.422189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.423281 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.423302 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.424006 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.424011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.424154 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.424213 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:10.425056 (+ 843us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:10.425076 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:47:10.425153 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:47:10.425300 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 22:47:10.425630 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.419801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.97.2:32829)
0927 22:47:10.420457 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.420461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.420479 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:10.420552 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.420556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.420633 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.420757 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:10.421138 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.422066 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.423415 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.423872 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.423975 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.424416 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:10.424959 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:10.425081 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.425199 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.425271 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":366,"threads_started":1}
I20260927 22:47:10.430174 1667 raft_consensus.cc:1261] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Refusing update from remote peer c61ea5fd47404af18eb8ef9a5067fd30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:10.430289 1945 raft_consensus.cc:1261] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Refusing update from remote peer c61ea5fd47404af18eb8ef9a5067fd30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:10.430949 2062 consensus_queue.cc:1035] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:10.431250 2022 consensus_queue.cc:1035] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:10.434399 2021 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: TxnStatusTablet state changed. Reason: New leader c61ea5fd47404af18eb8ef9a5067fd30. Latest consensus state: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:10.434648 2021 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:10.437299 2061 mvcc.cc:205] Tried to move back new op lower bound from 7334089647835111424 to 7334089647545180160. Current Snapshot: MvccSnapshot[applied={T|T < 7334089647835111424}]
I20260927 22:47:10.449162 2021 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:10.449476 2021 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P 733cc2855c594792af21fa0069f44092: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:10.459353 2020 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: TxnStatusTablet state changed. Reason: New leader c61ea5fd47404af18eb8ef9a5067fd30. Latest consensus state: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:10.459683 2020 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:10.462774 2022 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:10.463083 2022 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:10.464172 2022 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:10.464387 2022 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P c61ea5fd47404af18eb8ef9a5067fd30: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:10.472800 2020 tablet_replica.cc:439] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b2f552e67a4edaa9a9c335a31aff2d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42371 } } peers { permanent_uuid: "c61ea5fd47404af18eb8ef9a5067fd30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "733cc2855c594792af21fa0069f44092" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36395 } } }
I20260927 22:47:10.473083 2020 tablet_replica.cc:442] T 80667ef84ed14590ad1ef0a4a9b8fe15 P f3b2f552e67a4edaa9a9c335a31aff2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:10.506251 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.497058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60835 (local address 127.0.97.2:32829)
0927 22:47:10.497257 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.497264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.499984 (+ 2720us) server_negotiation.cc:407] Connection header received
0927 22:47:10.500274 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.500279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.500368 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.500429 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:10.501093 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.502201 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.504166 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.504798 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.504898 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.505488 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:10.505616 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:10.505679 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.505901 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.505984 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:47:10.506991 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.496861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32829 (local address 127.0.97.3:60835)
0927 22:47:10.499885 (+ 3024us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:10.499904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:10.500090 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:10.500617 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:10.500622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:10.500635 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:10.500922 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.500932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.502374 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.502379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.504010 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.504025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.505050 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.505056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.505230 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.505298 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:10.505752 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:10.505761 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:47:10.506493 (+ 732us) client_negotiation.cc:240] Negotiation successful
0927 22:47:10.506634 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2847,"thread_start_us":2706,"threads_started":1}
I20260927 22:47:10.552198 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.546101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35713 (local address 127.0.97.3:36395)
0927 22:47:10.546267 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.546272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.546784 (+ 512us) server_negotiation.cc:407] Connection header received
0927 22:47:10.546970 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.546975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.547043 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.547113 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:10.547684 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.548743 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.550384 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.550996 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.551124 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.551534 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:10.551657 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:10.551745 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.551891 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.551960 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:47:10.552363 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.546097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36395 (local address 127.0.97.2:35713)
0927 22:47:10.546689 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:10.546704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:10.546850 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:10.547138 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:10.547141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:10.547153 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:10.547497 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.547509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.548863 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.548867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.550258 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:10.550270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.551184 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.551189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:10.551306 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.551360 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:10.551761 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:10.551773 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:47:10.551872 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:47:10.552000 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":306,"threads_started":1}
I20260927 22:47:10.582197 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:10.582456 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:10.585320 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd9cd3c3b98a40a7b77c06f1b828688a: Sending DeleteTablet for 3 replicas of tablet 2abd0969eb684e238e1a665ce8f97899
I20260927 22:47:10.586020 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd0969eb684e238e1a665ce8f97899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47906
I20260927 22:47:10.586117 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd0969eb684e238e1a665ce8f97899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57636
I20260927 22:47:10.586272 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd0969eb684e238e1a665ce8f97899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56012
I20260927 22:47:10.597043 2084 tablet_replica.cc:330] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: stopping tablet replica
I20260927 22:47:10.597209 2082 tablet_replica.cc:330] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: stopping tablet replica
I20260927 22:47:10.597399 2084 raft_consensus.cc:2229] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:10.597508 2082 raft_consensus.cc:2229] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:10.597782 2082 raft_consensus.cc:2258] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:10.597884 2084 raft_consensus.cc:2258] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:10.598672 2083 tablet_replica.cc:330] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: stopping tablet replica
I20260927 22:47:10.598940 2083 raft_consensus.cc:2229] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:10.599221 2084 ts_tablet_manager.cc:1905] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:10.599361 2083 raft_consensus.cc:2258] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:10.600396 2083 ts_tablet_manager.cc:1905] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:10.606133 2084 ts_tablet_manager.cc:1918] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:10.606915 2082 ts_tablet_manager.cc:1905] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:10.608217 2083 ts_tablet_manager.cc:1918] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:10.608419 2083 log.cc:1196] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/2abd0969eb684e238e1a665ce8f97899
I20260927 22:47:10.608906 2083 ts_tablet_manager.cc:1939] T 2abd0969eb684e238e1a665ce8f97899 P c61ea5fd47404af18eb8ef9a5067fd30: Deleting consensus metadata
I20260927 22:47:10.610108 1495 catalog_manager.cc:4841] TS c61ea5fd47404af18eb8ef9a5067fd30 (127.0.97.2:32829): tablet 2abd0969eb684e238e1a665ce8f97899 (table test-table [id=29a60bef9b9a4fed9c5bbbe4988235ca]) successfully deleted
I20260927 22:47:10.611598 2084 log.cc:1196] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/2abd0969eb684e238e1a665ce8f97899
I20260927 22:47:10.612094 2084 ts_tablet_manager.cc:1939] T 2abd0969eb684e238e1a665ce8f97899 P 733cc2855c594792af21fa0069f44092: Deleting consensus metadata
I20260927 22:47:10.613201 1497 catalog_manager.cc:4841] TS 733cc2855c594792af21fa0069f44092 (127.0.97.3:36395): tablet 2abd0969eb684e238e1a665ce8f97899 (table test-table [id=29a60bef9b9a4fed9c5bbbe4988235ca]) successfully deleted
I20260927 22:47:10.613711 2082 ts_tablet_manager.cc:1918] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:10.613844 2082 log.cc:1196] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/2abd0969eb684e238e1a665ce8f97899
I20260927 22:47:10.614338 2082 ts_tablet_manager.cc:1939] T 2abd0969eb684e238e1a665ce8f97899 P f3b2f552e67a4edaa9a9c335a31aff2d: Deleting consensus metadata
I20260927 22:47:10.615329 1496 catalog_manager.cc:4841] TS f3b2f552e67a4edaa9a9c335a31aff2d (127.0.97.1:42371): tablet 2abd0969eb684e238e1a665ce8f97899 (table test-table [id=29a60bef9b9a4fed9c5bbbe4988235ca]) successfully deleted
Sep 27 22:47:10 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:10.628446 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.599282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.0.97.1:42371)
0927 22:47:10.599564 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.599570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.613527 (+ 13957us) server_negotiation.cc:407] Connection header received
0927 22:47:10.613731 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.613736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.613816 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.613920 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:10.615663 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.616696 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.617846 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.618432 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.618535 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.622062 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:10.622076 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:10.622093 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:10.622125 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:10.625455 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.626060 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.626069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.626073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.626216 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.626492 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.626497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.626500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.626903 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:10.627184 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.627335 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.627588 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260927 22:47:10.629117 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47908
I20260927 22:47:10.648689 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.631719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.97.62:43611)
0927 22:47:10.635878 (+ 4159us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.635882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.635898 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:47:10.635946 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.635949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.636015 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.636100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:10.637845 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.638576 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.640271 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.640691 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.640785 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.643461 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:10.643468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:10.643473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:10.643512 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:10.646320 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.647029 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.647033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.647035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.647154 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.647517 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.647520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.647522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.647771 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:10.647934 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.648252 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.648405 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4044,"thread_start_us":3882,"threads_started":1}
W20260927 22:47:10.649290 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58154
I20260927 22:47:10.650990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1582
I20260927 22:47:10.663177 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.651987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.0.97.1:42371)
0927 22:47:10.652152 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.652158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.652261 (+ 103us) server_negotiation.cc:407] Connection header received
0927 22:47:10.652411 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.652418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.652485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.652568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:10.653608 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.654284 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.655404 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.655874 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.655994 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.658380 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:10.658387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:10.658391 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:10.658421 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:10.660760 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.661328 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.661334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.661337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.661451 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.661775 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.661780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.661784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.662041 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:10.662274 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.662577 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.662812 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:47:10.664462 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47912
W20260927 22:47:10.665118 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:10.665295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 1582
I20260927 22:47:10.686791 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1721
Sep 27 22:47:10 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:10.703023 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.687593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.0.97.2:32829)
0927 22:47:10.687775 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.687779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.687797 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:10.687939 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.687944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.688017 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.688101 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:10.689596 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.690352 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.692066 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.692516 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.692605 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.697328 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:10.697338 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:10.697354 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:10.697388 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:10.700446 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.701162 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.701169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.701171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.701301 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.701670 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.701673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.701675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.701996 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:10.702241 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.702556 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.702711 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 22:47:10.703728 1851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57648
W20260927 22:47:10.704337 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:10.704514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 1721
I20260927 22:47:10.719955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1860
Sep 27 22:47:10 dist-test-slave-kr25 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:10.733757 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:10.720581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.0.97.3:36395)
0927 22:47:10.720747 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:10.720753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:10.720770 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:47:10.720916 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:10.720921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:10.720987 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:10.721074 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:10.722266 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.723058 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.724278 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:10.724684 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:10.724787 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:10.728613 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:10.728627 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:10.728645 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:10.728677 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:10.731172 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.731894 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.731901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.731904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.732025 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:10.732365 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:10.732368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:10.732371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:10.732695 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:10.732963 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:10.733249 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 22:47:10.733446 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:47:10.734381 1990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56018
W20260927 22:47:10.734917 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:10.735049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 1860
I20260927 22:47:10.751168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 1479
I20260927 22:47:10.752238 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:10.861020 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 1479
2026-09-27T22:47:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4469 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:10 dist-test-slave-kr25 krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:10 dist-test-slave-kr25 krb5kdc[2094](info): set up 2 sockets
Sep 27 22:47:10 dist-test-slave-kr25 krb5kdc[2094](info): commencing operation
krb5kdc: starting...
W20260927 22:47:11.851243 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:11 dist-test-slave-kr25 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:12.032709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46139
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:12.330996 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:12.331290 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:12.331421 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:12.338441 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:12.338537 2107 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:47:12.338584 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:12.338641 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:12.338688 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:12.338744 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:12.348430 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46139
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:12.349603 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:12.350832 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:12.362192 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:12.362232 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:12.362258 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:12.363034 2107 server_base.cc:1034] running on GCE node
I20260927 22:47:12.363776 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:12.365062 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:12.366261 2107 hybrid_clock.cc:648] HybridClock initialized: now 1790549232366210 us; error 33 us; skew 500 ppm
I20260927 22:47:12.366564 2107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:12 dist-test-slave-kr25 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:12.369405 2107 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:12.370733 2107 webserver.cc:460] Webserver started at http://127.0.97.62:38165/ using document root <none> and password file <none>
I20260927 22:47:12.371385 2107 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:12.371503 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:12.371760 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:12.374475 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "2d060e8516a249eda6e81288fc4db556"
format_stamp: "Formatted at 2026-09-27 22:47:12 on dist-test-slave-kr25"
server_key: "2cf838bcd44aa70400df287b3b26085f"
server_key_iv: "3b988a1f6ab6b91fbe13bf75e73d4dea"
server_key_version: "encryptionkey@0"
I20260927 22:47:12.375152 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "2d060e8516a249eda6e81288fc4db556"
format_stamp: "Formatted at 2026-09-27 22:47:12 on dist-test-slave-kr25"
server_key: "2cf838bcd44aa70400df287b3b26085f"
server_key_iv: "3b988a1f6ab6b91fbe13bf75e73d4dea"
server_key_version: "encryptionkey@0"
I20260927 22:47:12.379791 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:47:12.382833 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:12.384065 2107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:12.384200 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "2d060e8516a249eda6e81288fc4db556"
format_stamp: "Formatted at 2026-09-27 22:47:12 on dist-test-slave-kr25"
server_key: "2cf838bcd44aa70400df287b3b26085f"
server_key_iv: "3b988a1f6ab6b91fbe13bf75e73d4dea"
server_key_version: "encryptionkey@0"
I20260927 22:47:12.384380 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:12.406821 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:12.410496 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:12.410725 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:12.429783 2107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46139
I20260927 22:47:12.429814 2183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46139 every 8 connection(s)
I20260927 22:47:12.431053 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:12.434638 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:12.436658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 2107
I20260927 22:47:12.436867 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:12.437618 388 external_mini_cluster.cc:1428] Setting key 06d21296fe608d2e2af50251110c2275
I20260927 22:47:12.446964 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Bootstrap starting.
I20260927 22:47:12.450420 2184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:12.451339 2184 log.cc:824] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:47:12 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:12.454473 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: No bootstrap required, opened a new log
I20260927 22:47:12.460387 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:12.439721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.0.97.62:46139)
0927 22:47:12.440497 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:12.440511 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:12.440550 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:47:12.442153 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:12.442176 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:12.442549 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:12.442865 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:12.444490 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.446386 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.447676 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.448683 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.448871 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:12.452592 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:12.452602 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:12.452621 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:12.452660 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:12.456225 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:12.456864 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:12.456873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:12.456881 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:12.457069 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:12.457393 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:12.457398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:12.457402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:12.458055 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:12.458335 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:12.458637 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 22:47:12.459038 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":365,"threads_started":1}
I20260927 22:47:12.463975 2184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d060e8516a249eda6e81288fc4db556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46139 } }
I20260927 22:47:12.464294 2184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:12.464365 2184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d060e8516a249eda6e81288fc4db556, State: Initialized, Role: FOLLOWER
I20260927 22:47:12.464840 2184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d060e8516a249eda6e81288fc4db556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46139 } }
I20260927 22:47:12.465044 2184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:12.465135 2184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:12.465248 2184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:12.467182 2184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d060e8516a249eda6e81288fc4db556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46139 } }
I20260927 22:47:12.467649 2184 leader_election.cc:302] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d060e8516a249eda6e81288fc4db556; no voters:
I20260927 22:47:12.468130 2184 leader_election.cc:288] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:12.468250 2189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:12.469892 2189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [term 1 LEADER]: Becoming Leader. State: Replica: 2d060e8516a249eda6e81288fc4db556, State: Running, Role: LEADER
I20260927 22:47:12.470043 2184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:12.470382 2189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d060e8516a249eda6e81288fc4db556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46139 } }
I20260927 22:47:12.473081 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d060e8516a249eda6e81288fc4db556. Latest consensus state: current_term: 1 leader_uuid: "2d060e8516a249eda6e81288fc4db556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d060e8516a249eda6e81288fc4db556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46139 } } }
I20260927 22:47:12.473105 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d060e8516a249eda6e81288fc4db556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d060e8516a249eda6e81288fc4db556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46139 } } }
I20260927 22:47:12.473496 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:12.473505 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:12.475513 2198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:12.480625 2198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:12.488965 2198 catalog_manager.cc:1347] Generated new cluster ID: 4813f90475034480870d13c7aeee523d
I20260927 22:47:12.489070 2198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:12.498514 2198 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:12.500301 2198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:12.518801 2198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Generated new TSK 0
I20260927 22:47:12.519963 2198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:12.564409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46139
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:12.872715 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:12.873020 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:12.873183 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:12.880599 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:12.880733 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:12.880874 2210 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:47:12.880939 2210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:12.881040 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:12.890909 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:12.892271 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:12.893554 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:12.905447 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:12.905476 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:12.905452 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:12.906502 2210 server_base.cc:1034] running on GCE node
I20260927 22:47:12.907027 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:12.907928 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:12.909121 2210 hybrid_clock.cc:648] HybridClock initialized: now 1790549232909068 us; error 46 us; skew 500 ppm
I20260927 22:47:12.909404 2210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:12 dist-test-slave-kr25 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:12.917169 2210 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:12.918465 2210 webserver.cc:460] Webserver started at http://127.0.97.1:39287/ using document root <none> and password file <none>
I20260927 22:47:12.919098 2210 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:12.919191 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:12.919461 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:12.922258 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "9880e9c3212448c996f38b6864b10f3d"
format_stamp: "Formatted at 2026-09-27 22:47:12 on dist-test-slave-kr25"
server_key: "dd7ffa78661cfc6f6a2ba121adc9394d"
server_key_iv: "ab54f439c611fab0f5d08b09c1b64bf3"
server_key_version: "encryptionkey@0"
I20260927 22:47:12.922937 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9880e9c3212448c996f38b6864b10f3d"
format_stamp: "Formatted at 2026-09-27 22:47:12 on dist-test-slave-kr25"
server_key: "dd7ffa78661cfc6f6a2ba121adc9394d"
server_key_iv: "ab54f439c611fab0f5d08b09c1b64bf3"
server_key_version: "encryptionkey@0"
I20260927 22:47:12.927597 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260927 22:47:12.930634 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:12.931880 2210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:12.932015 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "9880e9c3212448c996f38b6864b10f3d"
format_stamp: "Formatted at 2026-09-27 22:47:12 on dist-test-slave-kr25"
server_key: "dd7ffa78661cfc6f6a2ba121adc9394d"
server_key_iv: "ab54f439c611fab0f5d08b09c1b64bf3"
server_key_version: "encryptionkey@0"
I20260927 22:47:12.932166 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:12.946267 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:12.950026 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:12.950238 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:12.952567 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:12.952678 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:12.952826 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:12.952919 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:47:12 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:12.979708 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:12.953465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46139 (local address 127.0.97.1:55993)
0927 22:47:12.954287 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:12.954339 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:12.956169 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:12.956900 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:12.956909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:12.957569 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:12.958446 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:12.958464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.959456 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.959462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:12.961687 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:12.961701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.962849 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.962855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:12.963090 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:12.967909 (+ 4819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:12.967940 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:12.970828 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:12.974688 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:12.974696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:12.974718 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:12.975333 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:12.975803 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:12.975808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:12.975811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:12.976052 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:12.976696 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:12.976706 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:12.977145 (+ 439us) client_negotiation.cc:769] Sending connection context
0927 22:47:12.977489 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 22:47:12.978025 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":343,"threads_started":1}
I20260927 22:47:12.980191 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:12.952977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55993 (local address 127.0.97.62:46139)
0927 22:47:12.953160 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:12.953166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:12.954459 (+ 1293us) server_negotiation.cc:407] Connection header received
0927 22:47:12.956459 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:12.956465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:12.956546 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:12.956658 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:12.958620 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.959294 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.961850 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.962460 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.962553 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:12.971093 (+ 8540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:12.971101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:12.971106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:12.971147 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:12.974518 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:12.975504 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:12.975509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:12.975513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:12.975678 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:12.976166 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:12.976172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:12.976176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:12.976547 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:12.976693 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:12.979624 (+ 2931us) server_negotiation.cc:299] Negotiation successful
0927 22:47:12.979906 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:47:13.001030 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:12.985783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55687 (local address 127.0.97.62:46139)
0927 22:47:12.985951 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:12.985957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:12.986886 (+ 929us) server_negotiation.cc:407] Connection header received
0927 22:47:12.987049 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:12.987055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:12.987129 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:12.987221 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:12.988565 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.989626 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.991811 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.992414 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.992541 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:12.994791 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:12.994798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:12.994803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:12.994837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:12.998143 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:12.999015 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:12.999021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:12.999024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:12.999156 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:12.999516 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:12.999519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:12.999521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:12.999838 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:12.999979 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:13.000472 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 22:47:13.000705 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:47:13.001050 2210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40701
I20260927 22:47:13.001278 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:12.985893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46139 (local address 127.0.97.1:55687)
0927 22:47:12.986796 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:12.986813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:12.986934 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:12.987268 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:12.987274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:12.987955 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:12.988408 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:12.988423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.989772 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.989778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:12.991647 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:12.991664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:12.992625 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:12.992632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:12.992821 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:12.993497 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:12.993518 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:12.994610 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:12.998278 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:12.998283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:12.998288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:12.998886 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:12.999255 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:12.999259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:12.999263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:12.999411 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:12.999964 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:12.999969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:13.000187 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 22:47:13.000345 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 22:47:13.000704 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":591,"threads_started":1}
I20260927 22:47:13.001564 2344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40701 every 8 connection(s)
I20260927 22:47:13.002647 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:13.004588 2345 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46139
I20260927 22:47:13.005214 2345 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:13.005810 2345 heartbeater.cc:511] Master 127.0.97.62:46139 requested a full tablet report, sending...
I20260927 22:47:13.007594 2139 ts_manager.cc:194] Registered new tserver with Master: 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701)
I20260927 22:47:13.008961 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55993
I20260927 22:47:13.011323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 2210
I20260927 22:47:13.011510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:13.012086 388 external_mini_cluster.cc:1428] Setting key f755d0524c36d64540018b0b87e31367
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:13.046717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46139
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:13.348122 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:13.348388 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:13.348510 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:13.355592 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:13.355697 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:13.355808 2349 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:47:13.355866 2349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:13.355962 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:13.365351 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:13.366534 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:13.367776 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:13.379734 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:13.379701 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:13.379654 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:13.380921 2349 server_base.cc:1034] running on GCE node
I20260927 22:47:13.381446 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:13.382254 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:13.383476 2349 hybrid_clock.cc:648] HybridClock initialized: now 1790549233383402 us; error 36 us; skew 500 ppm
I20260927 22:47:13.383772 2349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:13 dist-test-slave-kr25 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:13.386516 2349 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:13.387799 2349 webserver.cc:460] Webserver started at http://127.0.97.2:41033/ using document root <none> and password file <none>
I20260927 22:47:13.388449 2349 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:13.388556 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:13.388820 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:13.391978 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "a072f007b74b41a2b1ed35bc6b456db9"
format_stamp: "Formatted at 2026-09-27 22:47:13 on dist-test-slave-kr25"
server_key: "9626ef0b3fc573ac9620eb163e7d3d0c"
server_key_iv: "aaec7c50a9494eef22c6c4f646e9bb12"
server_key_version: "encryptionkey@0"
I20260927 22:47:13.392777 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a072f007b74b41a2b1ed35bc6b456db9"
format_stamp: "Formatted at 2026-09-27 22:47:13 on dist-test-slave-kr25"
server_key: "9626ef0b3fc573ac9620eb163e7d3d0c"
server_key_iv: "aaec7c50a9494eef22c6c4f646e9bb12"
server_key_version: "encryptionkey@0"
I20260927 22:47:13.397809 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:47:13.401157 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:13.402455 2349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:47:13.402595 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "a072f007b74b41a2b1ed35bc6b456db9"
format_stamp: "Formatted at 2026-09-27 22:47:13 on dist-test-slave-kr25"
server_key: "9626ef0b3fc573ac9620eb163e7d3d0c"
server_key_iv: "aaec7c50a9494eef22c6c4f646e9bb12"
server_key_version: "encryptionkey@0"
I20260927 22:47:13.402769 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:13.442334 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:13.446600 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:13.446869 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:13.449424 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:13.449536 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:13.449643 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:13.449724 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:47:13 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:13.471397 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.449971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50643 (local address 127.0.97.62:46139)
0927 22:47:13.450163 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:13.450170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:13.451565 (+ 1395us) server_negotiation.cc:407] Connection header received
0927 22:47:13.453509 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:13.453515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:13.453595 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:13.453679 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:13.455726 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.456849 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.459098 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.459685 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.459807 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.464578 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:13.464584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:13.464590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:13.464630 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:13.467889 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.468747 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.468753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.468757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.468879 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.469377 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.469382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.469385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.469748 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:13.469902 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:13.470765 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 22:47:13.471015 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:47:13.472245 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.450634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46139 (local address 127.0.97.2:50643)
0927 22:47:13.451385 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:13.451425 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:13.453253 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:13.453928 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:13.453939 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:13.454660 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:13.455494 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.455555 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.456992 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.456998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.458961 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.458973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.459945 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.459952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.460264 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.461089 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:13.461125 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:13.464329 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:13.468029 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.468038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.468058 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.468628 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.468979 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.468984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.468989 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.469229 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.469904 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:13.469916 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:13.470373 (+ 457us) client_negotiation.cc:769] Sending connection context
0927 22:47:13.470642 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 22:47:13.471009 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":330,"threads_started":1}
I20260927 22:47:13.490653 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.477825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46139 (local address 127.0.97.2:50135)
0927 22:47:13.478456 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:13.478482 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:13.478604 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:13.478909 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:13.478911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:13.479314 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:13.479671 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.479684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.480718 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.480722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.481719 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.481730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.482544 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.482550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.482688 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.483347 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:13.483374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:13.484376 (+ 1002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:13.487593 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.487598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.487607 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.488293 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.488783 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.488789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.488792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.488959 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.489661 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:13.489665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:13.489860 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 22:47:13.490057 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 22:47:13.490281 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":322,"threads_started":1}
I20260927 22:47:13.490877 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.477836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50135 (local address 127.0.97.62:46139)
0927 22:47:13.478053 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:13.478059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:13.478565 (+ 506us) server_negotiation.cc:407] Connection header received
0927 22:47:13.478704 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:13.478710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:13.478792 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:13.478900 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:13.479809 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.480591 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.481862 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.482358 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.482459 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.484606 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:13.484614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:13.484619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:13.484659 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:13.487472 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.488475 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.488481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.488485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.488667 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.489124 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.489128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.489130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.489483 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:13.489615 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:13.490172 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 22:47:13.490431 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:47:13.495257 2349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43843
I20260927 22:47:13.495292 2483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43843 every 8 connection(s)
I20260927 22:47:13.496505 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:13.498354 2484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46139
I20260927 22:47:13.498564 2484 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:13.499133 2484 heartbeater.cc:511] Master 127.0.97.62:46139 requested a full tablet report, sending...
I20260927 22:47:13.500349 2139 ts_manager.cc:194] Registered new tserver with Master: a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843)
I20260927 22:47:13.500965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 2349
I20260927 22:47:13.501119 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:13.501366 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50643
I20260927 22:47:13.501631 388 external_mini_cluster.cc:1428] Setting key bc0cc52115ef5986bc0ac13c14571726
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:13.536355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46139
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:13.838358 2488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:13.838622 2488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:13.838752 2488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:13.845723 2488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:13.845839 2488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:13.845953 2488 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:47:13.846019 2488 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:13.846114 2488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:13.855307 2488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:13.856477 2488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:13.857690 2488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:13.869524 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:13.869423 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:13.869450 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:13.870491 2488 server_base.cc:1034] running on GCE node
I20260927 22:47:13.871075 2488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:13.872015 2488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:13.873211 2488 hybrid_clock.cc:648] HybridClock initialized: now 1790549233873166 us; error 36 us; skew 500 ppm
I20260927 22:47:13.873539 2488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:13 dist-test-slave-kr25 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:13.876725 2488 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:13.878094 2488 webserver.cc:460] Webserver started at http://127.0.97.3:45301/ using document root <none> and password file <none>
I20260927 22:47:13.878749 2488 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:13.878861 2488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:13.879135 2488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:13.881953 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "ff78a6ec81ea45a78e931519bdacda89"
format_stamp: "Formatted at 2026-09-27 22:47:13 on dist-test-slave-kr25"
server_key: "38b2674859f1a41f2b4ac7f410109a03"
server_key_iv: "00e1c38fb877b0dfc2c31ee9e0bcee04"
server_key_version: "encryptionkey@0"
I20260927 22:47:13.882691 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ff78a6ec81ea45a78e931519bdacda89"
format_stamp: "Formatted at 2026-09-27 22:47:13 on dist-test-slave-kr25"
server_key: "38b2674859f1a41f2b4ac7f410109a03"
server_key_iv: "00e1c38fb877b0dfc2c31ee9e0bcee04"
server_key_version: "encryptionkey@0"
I20260927 22:47:13.887578 2488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 22:47:13.890772 2503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:13.892076 2488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:47:13.892211 2488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "ff78a6ec81ea45a78e931519bdacda89"
format_stamp: "Formatted at 2026-09-27 22:47:13 on dist-test-slave-kr25"
server_key: "38b2674859f1a41f2b4ac7f410109a03"
server_key_iv: "00e1c38fb877b0dfc2c31ee9e0bcee04"
server_key_version: "encryptionkey@0"
I20260927 22:47:13.892383 2488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:13.907290 2488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:13.910725 2488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:13.910961 2488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:13.913321 2488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:13.913414 2488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:13.913523 2488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:13.913596 2488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:47:13 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:13.938102 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.914432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46139 (local address 127.0.97.3:60735)
0927 22:47:13.915256 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:13.915303 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:13.917126 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:13.917828 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:13.917841 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:13.918569 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:13.919653 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.919688 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.921041 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.921048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.923501 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.923519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.924556 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.924563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.924845 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.926020 (+ 1175us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:13.926051 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:13.929185 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:13.932928 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.932942 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.932959 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.933543 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.933922 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.933928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.933932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.934226 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.934928 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:13.934941 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:13.935494 (+ 553us) client_negotiation.cc:769] Sending connection context
0927 22:47:13.935865 (+ 371us) client_negotiation.cc:240] Negotiation successful
0927 22:47:13.936453 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":352,"threads_started":1}
I20260927 22:47:13.939229 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.913905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60735 (local address 127.0.97.62:46139)
0927 22:47:13.914094 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:13.914101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:13.915423 (+ 1322us) server_negotiation.cc:407] Connection header received
0927 22:47:13.917386 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:13.917391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:13.917476 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:13.917575 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:13.919860 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.920875 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.923660 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.924270 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.924428 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.929504 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:13.929511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:13.929516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:13.929557 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:13.932802 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.933665 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.933671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.933676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.933811 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.934350 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.934356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.934360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.934698 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:13.934847 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:13.938697 (+ 3850us) server_negotiation.cc:299] Negotiation successful
0927 22:47:13.938935 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:47:13.948859 2488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38223
I20260927 22:47:13.948859 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38223 every 8 connection(s)
I20260927 22:47:13.950080 2488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:13.950832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 2488
I20260927 22:47:13.951005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:13.951598 388 external_mini_cluster.cc:1428] Setting key 12984d6273db8e350160edde3a3ab029
I20260927 22:47:13.952780 2623 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46139
I20260927 22:47:13.953027 2623 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:13.953750 2623 heartbeater.cc:511] Master 127.0.97.62:46139 requested a full tablet report, sending...
I20260927 22:47:13.955338 2138 ts_manager.cc:194] Registered new tserver with Master: ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:13.956422 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60735
I20260927 22:47:13.958629 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.945602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46139 (local address 127.0.97.3:56069)
0927 22:47:13.946186 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:13.946206 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:13.946361 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:13.946692 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:13.946697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:13.947188 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:13.947521 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.947535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.948612 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.948615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.949840 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:13.949852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.950907 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.950915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:13.951132 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.951881 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:13.951906 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:13.952897 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:13.956193 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.956199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.956203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.956683 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.957053 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:13.957058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:13.957061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:13.957206 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:13.957725 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:13.957729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:13.957961 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 22:47:13.958078 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:47:13.958293 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":51,"thread_start_us":301,"threads_started":1}
I20260927 22:47:13.958745 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.945527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56069 (local address 127.0.97.62:46139)
0927 22:47:13.945683 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:13.945687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:13.946291 (+ 604us) server_negotiation.cc:407] Connection header received
0927 22:47:13.946481 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:13.946487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:13.946559 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:13.946642 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:13.947713 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.948451 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.949984 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.950518 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:13.950648 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:13.953060 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:13.953066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:13.953069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:13.953105 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:13.956072 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.956792 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.956797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.956800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.956948 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:13.957303 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:13.957307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:13.957310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:13.957615 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:13.957767 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:13.958197 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 22:47:13.958387 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:47:13.966687 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:47:13 dist-test-slave-kr25 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:47:13 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:14.009132 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:13.995761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.97.62:46139)
0927 22:47:13.995961 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:13.995967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:13.996297 (+ 330us) server_negotiation.cc:407] Connection header received
0927 22:47:13.996477 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:13.996482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:13.996552 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:13.996638 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:13.998000 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:13.998772 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.000011 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.000459 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.000579 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.004357 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:14.004362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:14.004366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:14.004402 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:14.006784 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:14.007378 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:14.007383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:14.007385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:14.007534 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:14.007872 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:14.007878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:14.007881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:14.008141 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:14.008375 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.008639 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.008847 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:14.011509 2345 heartbeater.cc:503] Master 127.0.97.62:46139 was elected leader, sending a full tablet report...
I20260927 22:47:14.013094 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:14.019586 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:14.043501 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.030406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40701 (local address 127.0.0.1:42422)
0927 22:47:14.031239 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.031287 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.031478 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.035127 (+ 3649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.035135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.035168 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:14.035658 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.035671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.037608 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.037614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.039995 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.040008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.042238 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.042245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.042701 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.042763 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.042872 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.043064 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":244,"spinlock_wait_cycles":4992,"thread_start_us":492,"threads_started":1}
I20260927 22:47:14.044375 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.030669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.0.97.1:40701)
0927 22:47:14.034483 (+ 3814us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.034493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.034523 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:47:14.034615 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.034623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.034884 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.035092 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:14.035837 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.037421 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.040187 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.041802 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.041913 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.042612 (+ 699us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.043902 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.044080 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3618,"thread_start_us":3487,"threads_started":1}
I20260927 22:47:14.045323 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.030609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38223 (local address 127.0.0.1:36290)
0927 22:47:14.035158 (+ 4549us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.035188 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.035307 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.036183 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.036188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.036212 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:14.036583 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.036595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.039611 (+ 3016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.039616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.042551 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.042564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.044561 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.044567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.044741 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.044772 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.044848 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.044961 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4424,"mutex_wait_us":4,"thread_start_us":4331,"threads_started":1}
I20260927 22:47:14.046921 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.031131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.0.97.3:38223)
0927 22:47:14.034843 (+ 3712us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.034852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.035418 (+ 566us) server_negotiation.cc:407] Connection header received
0927 22:47:14.035532 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.035551 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.035775 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.035941 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:14.036801 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.039424 (+ 2623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.042721 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.044083 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.044198 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.044416 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.045680 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.045837 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3549,"thread_start_us":3397,"threads_started":1}
I20260927 22:47:14.048987 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3396a6d5324669ac1ff22dff856f51 (DEFAULT_TABLE table=test-table [id=e801efba47814cc3870299ab8979b46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:14.050187 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3396a6d5324669ac1ff22dff856f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.049948 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3396a6d5324669ac1ff22dff856f51 (DEFAULT_TABLE table=test-table [id=e801efba47814cc3870299ab8979b46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:14.051265 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3396a6d5324669ac1ff22dff856f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.053427 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.035285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43843 (local address 127.0.0.1:43062)
0927 22:47:14.044988 (+ 9703us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.045004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.045103 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.047160 (+ 2057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.047164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.047192 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:14.047349 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.047358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.049204 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.049209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.050850 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.050864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.052474 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.052480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.052856 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.052889 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.052989 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.053133 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9620,"thread_start_us":11375,"threads_started":1}
I20260927 22:47:14.054226 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.035718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.0.97.2:43843)
0927 22:47:14.046580 (+ 10862us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.046588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.046616 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:47:14.046704 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.046711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.046948 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.047528 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:14.047723 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.049051 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.051028 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.052304 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.052437 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.052623 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.053774 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.053937 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10704,"thread_start_us":10587,"threads_started":1}
I20260927 22:47:14.057507 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3396a6d5324669ac1ff22dff856f51 (DEFAULT_TABLE table=test-table [id=e801efba47814cc3870299ab8979b46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:14.058754 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3396a6d5324669ac1ff22dff856f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.061589 2642 tablet_bootstrap.cc:492] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Bootstrap starting.
I20260927 22:47:14.063651 2643 tablet_bootstrap.cc:492] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Bootstrap starting.
I20260927 22:47:14.064954 2642 tablet_bootstrap.cc:654] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.066196 2642 log.cc:824] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:14.067015 2643 tablet_bootstrap.cc:654] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.068141 2643 log.cc:824] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:14.069803 2642 tablet_bootstrap.cc:492] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: No bootstrap required, opened a new log
I20260927 22:47:14.070075 2642 ts_tablet_manager.cc:1397] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260927 22:47:14.070868 2646 tablet_bootstrap.cc:492] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Bootstrap starting.
I20260927 22:47:14.071159 2643 tablet_bootstrap.cc:492] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: No bootstrap required, opened a new log
I20260927 22:47:14.071475 2643 ts_tablet_manager.cc:1397] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:47:14.073501 2646 tablet_bootstrap.cc:654] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.074458 2646 log.cc:824] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:14.076797 2646 tablet_bootstrap.cc:492] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: No bootstrap required, opened a new log
I20260927 22:47:14.077031 2646 ts_tablet_manager.cc:1397] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 22:47:14.080904 2642 raft_consensus.cc:348] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.081388 2642 raft_consensus.cc:374] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.081487 2642 raft_consensus.cc:729] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9880e9c3212448c996f38b6864b10f3d, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.081548 2643 raft_consensus.cc:348] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.081921 2643 raft_consensus.cc:374] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.082015 2643 raft_consensus.cc:729] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff78a6ec81ea45a78e931519bdacda89, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.082104 2642 consensus_queue.cc:260] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.082563 2643 consensus_queue.cc:260] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.084008 2642 ts_tablet_manager.cc:1428] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260927 22:47:14.084944 2623 heartbeater.cc:503] Master 127.0.97.62:46139 was elected leader, sending a full tablet report...
I20260927 22:47:14.085559 2643 ts_tablet_manager.cc:1428] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 22:47:14.086052 2646 raft_consensus.cc:348] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.086477 2646 raft_consensus.cc:374] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.086580 2646 raft_consensus.cc:729] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a072f007b74b41a2b1ed35bc6b456db9, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.087106 2646 consensus_queue.cc:260] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.088186 2484 heartbeater.cc:503] Master 127.0.97.62:46139 was elected leader, sending a full tablet report...
I20260927 22:47:14.088687 2646 ts_tablet_manager.cc:1428] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
W20260927 22:47:14.202451 2624 tablet.cc:2369] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:47:14.248358 2485 tablet.cc:2369] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:47:14.256150 2346 tablet.cc:2369] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:14.277613 2649 raft_consensus.cc:482] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:14.277895 2649 raft_consensus.cc:504] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.279482 2649 leader_election.cc:288] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843)
I20260927 22:47:14.281886 2650 raft_consensus.cc:482] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:14.282238 2650 raft_consensus.cc:504] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.283903 2650 leader_election.cc:288] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:14.286222 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.279915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40701 (local address 127.0.97.3:47143)
0927 22:47:14.280110 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.280131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.280253 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.280603 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.280608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.280644 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:14.280820 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.280829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.282057 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.282062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.283776 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.283790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.285483 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.285489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.285687 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.285721 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.285807 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.285934 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":21}
I20260927 22:47:14.286760 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.279963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47143 (local address 127.0.97.1:40701)
0927 22:47:14.280149 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.280154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.280174 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:47:14.280388 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.280393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.280481 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.280565 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:14.280991 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.281942 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.283978 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.285190 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.285305 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.285403 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.286406 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.286529 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:47:14.288312 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "ff78a6ec81ea45a78e931519bdacda89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d" is_pre_election: true
I20260927 22:47:14.288926 2300 raft_consensus.cc:2454] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff78a6ec81ea45a78e931519bdacda89 in term 0.
I20260927 22:47:14.287426 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.279956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41827 (local address 127.0.97.2:43843)
0927 22:47:14.280146 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.280151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.280917 (+ 766us) server_negotiation.cc:407] Connection header received
0927 22:47:14.281073 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.281078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.281151 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.281234 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:14.281690 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.282847 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.285171 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.286312 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.286414 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.286521 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.287027 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.287166 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:47:14.289757 2506 leader_election.cc:302] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, ff78a6ec81ea45a78e931519bdacda89; no voters:
I20260927 22:47:14.290216 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.279937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43843 (local address 127.0.97.3:41827)
0927 22:47:14.280822 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.280835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.280944 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.281247 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.281250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.281266 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:14.281552 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.281558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.282993 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.282998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.284982 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.284998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.286689 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.286694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.286858 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.286889 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.289704 (+ 2815us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.289833 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":216,"thread_start_us":363,"threads_started":1}
I20260927 22:47:14.290512 2649 raft_consensus.cc:2781] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:14.290882 2649 raft_consensus.cc:482] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:14.291148 2649 raft_consensus.cc:3037] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.291297 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "ff78a6ec81ea45a78e931519bdacda89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a072f007b74b41a2b1ed35bc6b456db9" is_pre_election: true
I20260927 22:47:14.291781 2434 raft_consensus.cc:2454] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff78a6ec81ea45a78e931519bdacda89 in term 0.
I20260927 22:47:14.294317 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.284433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56379 (local address 127.0.97.1:40701)
0927 22:47:14.287128 (+ 2695us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.287133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.287646 (+ 513us) server_negotiation.cc:407] Connection header received
0927 22:47:14.287895 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.287900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.287968 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.288031 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:14.288708 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.289707 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.292265 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.293444 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.293552 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.293639 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.293953 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.294066 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2606,"thread_start_us":6793,"threads_started":1}
I20260927 22:47:14.294360 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.284085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40701 (local address 127.0.97.2:56379)
0927 22:47:14.287563 (+ 3478us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.287579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.287752 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.288200 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.288204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.288228 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:14.288556 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.288569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.289978 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.289983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.292097 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.292112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.293619 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.293625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.293786 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.293814 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.293928 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.294057 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3331,"thread_start_us":3559,"threads_started":1}
I20260927 22:47:14.294309 2649 raft_consensus.cc:504] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.294945 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d" is_pre_election: true
I20260927 22:47:14.295149 2300 raft_consensus.cc:2454] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 0.
I20260927 22:47:14.295401 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.285590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52343 (local address 127.0.97.3:38223)
0927 22:47:14.285793 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.285799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.288012 (+ 2213us) server_negotiation.cc:407] Connection header received
0927 22:47:14.288284 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.288305 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.288373 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.288427 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:14.289171 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.290139 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.293387 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.294573 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.294676 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.294788 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.295063 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.295178 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:47:14.295482 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.284435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38223 (local address 127.0.97.2:52343)
0927 22:47:14.287919 (+ 3484us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.287937 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.288146 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.288637 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.288641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.288666 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:14.288996 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.289005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.290436 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.290440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.293205 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.293220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.294740 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.294745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.294922 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.294954 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.295053 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.295166 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3345,"thread_start_us":2816,"threads_started":1}
I20260927 22:47:14.295727 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "ff78a6ec81ea45a78e931519bdacda89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d"
I20260927 22:47:14.295674 2367 leader_election.cc:302] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, a072f007b74b41a2b1ed35bc6b456db9; no voters:
I20260927 22:47:14.295886 2300 raft_consensus.cc:3037] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.296059 2650 raft_consensus.cc:2781] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:14.296092 2649 leader_election.cc:288] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [CANDIDATE]: Term 1 election: Requested vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843)
I20260927 22:47:14.296176 2650 raft_consensus.cc:482] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:14.296264 2650 raft_consensus.cc:3037] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.296784 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "ff78a6ec81ea45a78e931519bdacda89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a072f007b74b41a2b1ed35bc6b456db9"
I20260927 22:47:14.296897 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89" is_pre_election: true
I20260927 22:47:14.297319 2573 raft_consensus.cc:2379] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a072f007b74b41a2b1ed35bc6b456db9 in current term 1: Already voted for candidate ff78a6ec81ea45a78e931519bdacda89 in this term.
I20260927 22:47:14.298894 2300 raft_consensus.cc:2454] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff78a6ec81ea45a78e931519bdacda89 in term 1.
I20260927 22:47:14.299105 2650 raft_consensus.cc:504] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.299239 2506 leader_election.cc:302] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, ff78a6ec81ea45a78e931519bdacda89; no voters:
I20260927 22:47:14.299481 2434 raft_consensus.cc:2379] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff78a6ec81ea45a78e931519bdacda89 in current term 1: Already voted for candidate a072f007b74b41a2b1ed35bc6b456db9 in this term.
I20260927 22:47:14.299522 2649 raft_consensus.cc:2781] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:14.300034 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d"
I20260927 22:47:14.300171 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3396a6d5324669ac1ff22dff856f51" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89"
I20260927 22:47:14.300290 2300 raft_consensus.cc:2379] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a072f007b74b41a2b1ed35bc6b456db9 in current term 1: Already voted for candidate ff78a6ec81ea45a78e931519bdacda89 in this term.
I20260927 22:47:14.300475 2650 leader_election.cc:288] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 election: Requested vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:14.305008 2649 raft_consensus.cc:686] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 1 LEADER]: Becoming Leader. State: Replica: ff78a6ec81ea45a78e931519bdacda89, State: Running, Role: LEADER
I20260927 22:47:14.305454 2649 consensus_queue.cc:237] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } }
I20260927 22:47:14.308885 2366 leader_election.cc:302] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a072f007b74b41a2b1ed35bc6b456db9; no voters: 9880e9c3212448c996f38b6864b10f3d, ff78a6ec81ea45a78e931519bdacda89
I20260927 22:47:14.309206 2650 raft_consensus.cc:2726] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 22:47:14.310016 2138 catalog_manager.cc:5495] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ff78a6ec81ea45a78e931519bdacda89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:47:14 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:14.357312 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.345578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46139 (local address 127.0.0.1:35072)
0927 22:47:14.346116 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.346131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.346247 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.346501 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.346505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.347130 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:14.347382 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.347390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.348310 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.348314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.349244 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.349254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.349856 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.349860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.349991 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.350561 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:14.350576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:14.352516 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:14.355258 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:14.355267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:14.355273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:14.355711 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:14.356012 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:14.356015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:14.356018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:14.356143 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:14.356552 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:14.356558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:14.356697 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.356817 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.356998 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":307,"threads_started":1}
I20260927 22:47:14.357363 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.345530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.0.97.62:46139)
0927 22:47:14.345702 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.345709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.346211 (+ 502us) server_negotiation.cc:407] Connection header received
0927 22:47:14.346334 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.346337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.346393 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.346486 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:14.347515 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.348201 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.349350 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.349730 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.349816 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.352715 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:14.352720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:14.352723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:14.352751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:14.355161 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:14.355806 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:14.355810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:14.355812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:14.355928 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:14.356229 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:14.356232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:14.356234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:14.356458 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:14.356568 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.356913 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.357094 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":55}
I20260927 22:47:14.361830 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35072:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:47:14.362785 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:14.368575 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 8056eb5fadcc4cceb6808d745cb17f3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4da3411b858c493b901ea589e73d131b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:14.368480 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 8056eb5fadcc4cceb6808d745cb17f3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4da3411b858c493b901ea589e73d131b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:14.368712 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 8056eb5fadcc4cceb6808d745cb17f3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4da3411b858c493b901ea589e73d131b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:14.369335 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8056eb5fadcc4cceb6808d745cb17f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.369360 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8056eb5fadcc4cceb6808d745cb17f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.369370 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8056eb5fadcc4cceb6808d745cb17f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.374600 2646 tablet_bootstrap.cc:492] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: Bootstrap starting.
I20260927 22:47:14.376006 2646 tablet_bootstrap.cc:654] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.376189 2642 tablet_bootstrap.cc:492] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: Bootstrap starting.
I20260927 22:47:14.376492 2643 tablet_bootstrap.cc:492] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: Bootstrap starting.
I20260927 22:47:14.377655 2642 tablet_bootstrap.cc:654] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.377871 2643 tablet_bootstrap.cc:654] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.378113 2646 tablet_bootstrap.cc:492] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: No bootstrap required, opened a new log
I20260927 22:47:14.378229 2646 ts_tablet_manager.cc:1397] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 22:47:14.378866 2646 raft_consensus.cc:348] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.379055 2646 raft_consensus.cc:374] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.379134 2646 raft_consensus.cc:729] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a072f007b74b41a2b1ed35bc6b456db9, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.379336 2646 consensus_queue.cc:260] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.379904 2650 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.380246 2650 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:14.379932 2646 ts_tablet_manager.cc:1428] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:47:14.381229 2642 tablet_bootstrap.cc:492] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: No bootstrap required, opened a new log
I20260927 22:47:14.381376 2642 ts_tablet_manager.cc:1397] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 22:47:14.382085 2642 raft_consensus.cc:348] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.382330 2642 raft_consensus.cc:374] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.382429 2642 raft_consensus.cc:729] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9880e9c3212448c996f38b6864b10f3d, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.382642 2642 consensus_queue.cc:260] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.383244 2642 ts_tablet_manager.cc:1428] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:47:14.383141 2648 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.383752 2648 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:14.386245 2643 tablet_bootstrap.cc:492] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: No bootstrap required, opened a new log
I20260927 22:47:14.386359 2643 ts_tablet_manager.cc:1397] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260927 22:47:14.386902 2643 raft_consensus.cc:348] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.387068 2643 raft_consensus.cc:374] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.387126 2643 raft_consensus.cc:729] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff78a6ec81ea45a78e931519bdacda89, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.387290 2643 consensus_queue.cc:260] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.387647 2649 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.387795 2643 ts_tablet_manager.cc:1428] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 22:47:14.387941 2649 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:14.423805 2650 raft_consensus.cc:482] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:14.424023 2650 raft_consensus.cc:504] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.424706 2650 leader_election.cc:288] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:14.425179 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056eb5fadcc4cceb6808d745cb17f3c" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89" is_pre_election: true
I20260927 22:47:14.425181 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056eb5fadcc4cceb6808d745cb17f3c" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d" is_pre_election: true
I20260927 22:47:14.425451 2300 raft_consensus.cc:2454] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 0.
I20260927 22:47:14.425465 2573 raft_consensus.cc:2454] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 0.
I20260927 22:47:14.425848 2367 leader_election.cc:302] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, a072f007b74b41a2b1ed35bc6b456db9, ff78a6ec81ea45a78e931519bdacda89; no voters:
I20260927 22:47:14.426123 2650 raft_consensus.cc:2781] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:14.426208 2650 raft_consensus.cc:482] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:14.426301 2650 raft_consensus.cc:3037] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.428378 2650 raft_consensus.cc:504] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.428879 2650 leader_election.cc:288] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 election: Requested vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:14.429284 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056eb5fadcc4cceb6808d745cb17f3c" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d"
I20260927 22:47:14.429334 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056eb5fadcc4cceb6808d745cb17f3c" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89"
I20260927 22:47:14.429478 2300 raft_consensus.cc:3037] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.429543 2573 raft_consensus.cc:3037] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.431383 2300 raft_consensus.cc:2454] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 1.
I20260927 22:47:14.431772 2367 leader_election.cc:302] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, a072f007b74b41a2b1ed35bc6b456db9; no voters:
I20260927 22:47:14.432015 2650 raft_consensus.cc:2781] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:14.432474 2573 raft_consensus.cc:2454] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 1.
I20260927 22:47:14.432473 2650 raft_consensus.cc:686] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [term 1 LEADER]: Becoming Leader. State: Replica: a072f007b74b41a2b1ed35bc6b456db9, State: Running, Role: LEADER
I20260927 22:47:14.432977 2650 consensus_queue.cc:237] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.436448 2666 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: TxnStatusTablet state changed. Reason: New leader a072f007b74b41a2b1ed35bc6b456db9. Latest consensus state: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.436843 2666 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:14.437263 2139 catalog_manager.cc:5495] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:14.438849 2668 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:47:14.439091 2668 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:47:14.466675 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.460515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.0.97.2:43843)
0927 22:47:14.460743 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.460749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.460826 (+ 77us) server_negotiation.cc:407] Connection header received
0927 22:47:14.461109 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.461115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.461218 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.461375 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:14.461856 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.462709 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.464565 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.465002 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.465136 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.465629 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.466105 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.466206 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.466374 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.466435 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 22:47:14.466838 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.460485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43843 (local address 127.0.0.1:43078)
0927 22:47:14.460721 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.460744 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.460900 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.461444 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.461450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.461464 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:14.461713 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.461725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.462843 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.462849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.464423 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.464439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.465152 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.465159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.465377 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.465464 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.466210 (+ 746us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.466229 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.466351 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.466499 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 22:47:14.474623 2300 raft_consensus.cc:1261] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Refusing update from remote peer a072f007b74b41a2b1ed35bc6b456db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:14.474623 2573 raft_consensus.cc:1261] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Refusing update from remote peer a072f007b74b41a2b1ed35bc6b456db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:14.475423 2666 consensus_queue.cc:1035] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:14.475747 2650 consensus_queue.cc:1035] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:14.486495 2648 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: TxnStatusTablet state changed. Reason: New leader a072f007b74b41a2b1ed35bc6b456db9. Latest consensus state: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.486853 2648 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:14.489894 2649 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: TxnStatusTablet state changed. Reason: New leader a072f007b74b41a2b1ed35bc6b456db9. Latest consensus state: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.490190 2649 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:14.498767 2666 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.499038 2666 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:14.501978 2666 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.502240 2666 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P a072f007b74b41a2b1ed35bc6b456db9: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:14.505263 2649 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.505519 2649 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P ff78a6ec81ea45a78e931519bdacda89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:14.510339 2648 tablet_replica.cc:439] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } } }
I20260927 22:47:14.510571 2648 tablet_replica.cc:442] T 8056eb5fadcc4cceb6808d745cb17f3c P 9880e9c3212448c996f38b6864b10f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:14.522010 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.516389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.0.97.3:38223)
0927 22:47:14.516559 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.516565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.517082 (+ 517us) server_negotiation.cc:407] Connection header received
0927 22:47:14.517269 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.517274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.517347 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.517467 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:14.517887 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.518635 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.519990 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.520422 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.520526 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.520919 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.521422 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.521552 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.521656 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.521730 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:47:14.537355 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.531662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43843 (local address 127.0.97.3:49669)
0927 22:47:14.532280 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.532296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.532402 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.532711 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.532713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.532722 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:14.532937 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.532943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.534276 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.534278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.535423 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.535467 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.536391 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.536395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.536500 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.536552 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.536949 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.536956 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.537006 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.537099 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":314,"threads_started":1}
I20260927 22:47:14.537364 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.531679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49669 (local address 127.0.97.2:43843)
0927 22:47:14.531863 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.531869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.532374 (+ 505us) server_negotiation.cc:407] Connection header received
0927 22:47:14.532506 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.532511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.532612 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.532707 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:14.533083 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.534163 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.535582 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.536232 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.536365 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.536737 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.536869 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.536954 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.537028 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.537100 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:47:14.546993 2300 raft_consensus.cc:1261] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Refusing update from remote peer ff78a6ec81ea45a78e931519bdacda89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:14.547188 2434 raft_consensus.cc:1261] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Refusing update from remote peer ff78a6ec81ea45a78e931519bdacda89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:14.547601 2655 consensus_queue.cc:1035] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:14.547940 2649 consensus_queue.cc:1035] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:14.555685 2672 mvcc.cc:205] Tried to move back new op lower bound from 7334089664698900480 to 7334089663718518784. Current Snapshot: MvccSnapshot[applied={T|T < 7334089664698900480}]
I20260927 22:47:14.579861 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.573639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33625 (local address 127.0.97.3:38223)
0927 22:47:14.573785 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.573791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.574477 (+ 686us) server_negotiation.cc:407] Connection header received
0927 22:47:14.574638 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.574643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.574710 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.574780 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:14.575370 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.576244 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.577771 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.578333 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.578419 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.579057 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.579175 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.579239 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.579493 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.579587 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:47:14.580479 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.573730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38223 (local address 127.0.97.2:33625)
0927 22:47:14.574375 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.574395 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.574523 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.574805 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.574810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.574821 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:14.575224 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.575237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.576393 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.576397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.577625 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.577640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.578623 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.578630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.578804 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.578869 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.579325 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.579334 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.580066 (+ 732us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.580199 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":353,"threads_started":1}
I20260927 22:47:14.619529 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:14.619792 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:14.623402 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Sending DeleteTablet for 3 replicas of tablet 9d3396a6d5324669ac1ff22dff856f51
I20260927 22:47:14.624433 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3396a6d5324669ac1ff22dff856f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42422
I20260927 22:47:14.624702 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3396a6d5324669ac1ff22dff856f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36290
I20260927 22:47:14.626694 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3396a6d5324669ac1ff22dff856f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43062
I20260927 22:47:14.631099 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:47:14.631901 2694 tablet_replica.cc:330] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: stopping tablet replica
I20260927 22:47:14.632038 2693 tablet_replica.cc:330] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: stopping tablet replica
I20260927 22:47:14.632299 2694 raft_consensus.cc:2229] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:14.632344 2693 raft_consensus.cc:2229] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 22:47:14.632366 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:14.632656 2693 raft_consensus.cc:2258] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:14.632884 2694 raft_consensus.cc:2258] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:14.634647 2694 ts_tablet_manager.cc:1905] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:14.640527 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 55a79710f7904a039af6e18522efb0c7 (DEFAULT_TABLE table=test-table [id=8f731c7a183344bd89667c3638fc0a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:14.640978 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a79710f7904a039af6e18522efb0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.642400 2694 ts_tablet_manager.cc:1918] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:14.642565 2694 log.cc:1196] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/9d3396a6d5324669ac1ff22dff856f51
I20260927 22:47:14.643096 2694 ts_tablet_manager.cc:1939] T 9d3396a6d5324669ac1ff22dff856f51 P ff78a6ec81ea45a78e931519bdacda89: Deleting consensus metadata
I20260927 22:47:14.644420 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 55a79710f7904a039af6e18522efb0c7 (DEFAULT_TABLE table=test-table [id=8f731c7a183344bd89667c3638fc0a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:14.644508 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 55a79710f7904a039af6e18522efb0c7 (DEFAULT_TABLE table=test-table [id=8f731c7a183344bd89667c3638fc0a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:14.644697 2124 catalog_manager.cc:4841] TS ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223): tablet 9d3396a6d5324669ac1ff22dff856f51 (table test-table [id=e801efba47814cc3870299ab8979b46c]) successfully deleted
I20260927 22:47:14.644819 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a79710f7904a039af6e18522efb0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.645069 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a79710f7904a039af6e18522efb0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:14.650878 2642 tablet_bootstrap.cc:492] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: Bootstrap starting.
I20260927 22:47:14.651304 2646 tablet_bootstrap.cc:492] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: Bootstrap starting.
I20260927 22:47:14.652488 2642 tablet_bootstrap.cc:654] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.652918 2646 tablet_bootstrap.cc:654] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.653434 2643 tablet_bootstrap.cc:492] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: Bootstrap starting.
I20260927 22:47:14.654948 2643 tablet_bootstrap.cc:654] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:14.656792 2693 ts_tablet_manager.cc:1905] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:14.662932 2695 tablet_replica.cc:330] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: stopping tablet replica
I20260927 22:47:14.663352 2695 raft_consensus.cc:2229] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:14.663727 2695 raft_consensus.cc:2258] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:14.673427 2695 ts_tablet_manager.cc:1905] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:14.677186 2693 ts_tablet_manager.cc:1918] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:14.677371 2693 log.cc:1196] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/9d3396a6d5324669ac1ff22dff856f51
I20260927 22:47:14.677846 2693 ts_tablet_manager.cc:1939] T 9d3396a6d5324669ac1ff22dff856f51 P 9880e9c3212448c996f38b6864b10f3d: Deleting consensus metadata
I20260927 22:47:14.679577 2125 catalog_manager.cc:4841] TS 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701): tablet 9d3396a6d5324669ac1ff22dff856f51 (table test-table [id=e801efba47814cc3870299ab8979b46c]) successfully deleted
I20260927 22:47:14.685376 2695 ts_tablet_manager.cc:1918] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:14.685539 2695 log.cc:1196] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/9d3396a6d5324669ac1ff22dff856f51
I20260927 22:47:14.686065 2695 ts_tablet_manager.cc:1939] T 9d3396a6d5324669ac1ff22dff856f51 P a072f007b74b41a2b1ed35bc6b456db9: Deleting consensus metadata
I20260927 22:47:14.687697 2124 catalog_manager.cc:4841] TS a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843): tablet 9d3396a6d5324669ac1ff22dff856f51 (table test-table [id=e801efba47814cc3870299ab8979b46c]) successfully deleted
I20260927 22:47:14.689708 2646 tablet_bootstrap.cc:492] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: No bootstrap required, opened a new log
I20260927 22:47:14.689888 2646 ts_tablet_manager.cc:1397] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.015s
I20260927 22:47:14.690387 2643 tablet_bootstrap.cc:492] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: No bootstrap required, opened a new log
I20260927 22:47:14.690667 2646 raft_consensus.cc:348] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.690907 2646 raft_consensus.cc:374] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.690995 2646 raft_consensus.cc:729] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a072f007b74b41a2b1ed35bc6b456db9, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.691321 2642 tablet_bootstrap.cc:492] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: No bootstrap required, opened a new log
I20260927 22:47:14.691282 2646 consensus_queue.cc:260] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.691998 2646 ts_tablet_manager.cc:1428] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:14.692415 2643 ts_tablet_manager.cc:1397] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.008s
I20260927 22:47:14.693214 2643 raft_consensus.cc:348] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.693466 2643 raft_consensus.cc:374] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.693554 2643 raft_consensus.cc:729] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff78a6ec81ea45a78e931519bdacda89, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.693780 2642 ts_tablet_manager.cc:1397] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.005s
I20260927 22:47:14.694449 2642 raft_consensus.cc:348] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.694692 2642 raft_consensus.cc:374] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:14.694770 2642 raft_consensus.cc:729] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9880e9c3212448c996f38b6864b10f3d, State: Initialized, Role: FOLLOWER
I20260927 22:47:14.695021 2642 consensus_queue.cc:260] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.695395 2643 consensus_queue.cc:260] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.696125 2642 ts_tablet_manager.cc:1428] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:47:14.697346 2643 ts_tablet_manager.cc:1428] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 22:47:14.797359 2666 raft_consensus.cc:482] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:14.797600 2666 raft_consensus.cc:504] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.798257 2666 leader_election.cc:288] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:14.798754 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a79710f7904a039af6e18522efb0c7" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d" is_pre_election: true
I20260927 22:47:14.798897 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a79710f7904a039af6e18522efb0c7" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89" is_pre_election: true
I20260927 22:47:14.798990 2299 raft_consensus.cc:2454] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 0.
I20260927 22:47:14.799113 2573 raft_consensus.cc:2454] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 0.
I20260927 22:47:14.799568 2366 leader_election.cc:302] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a072f007b74b41a2b1ed35bc6b456db9, ff78a6ec81ea45a78e931519bdacda89; no voters:
I20260927 22:47:14.799837 2666 raft_consensus.cc:2781] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:14.799932 2666 raft_consensus.cc:482] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:14.800040 2666 raft_consensus.cc:3037] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.802163 2666 raft_consensus.cc:504] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.802650 2666 leader_election.cc:288] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 election: Requested vote from peers 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:14.803128 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a79710f7904a039af6e18522efb0c7" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89"
I20260927 22:47:14.803274 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a79710f7904a039af6e18522efb0c7" candidate_uuid: "a072f007b74b41a2b1ed35bc6b456db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880e9c3212448c996f38b6864b10f3d"
I20260927 22:47:14.803339 2573 raft_consensus.cc:3037] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.803462 2299 raft_consensus.cc:3037] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:14.806267 2573 raft_consensus.cc:2454] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 1.
I20260927 22:47:14.806272 2299 raft_consensus.cc:2454] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a072f007b74b41a2b1ed35bc6b456db9 in term 1.
I20260927 22:47:14.806686 2367 leader_election.cc:302] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, a072f007b74b41a2b1ed35bc6b456db9; no voters:
I20260927 22:47:14.807071 2666 raft_consensus.cc:2781] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:14.807215 2666 raft_consensus.cc:686] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 LEADER]: Becoming Leader. State: Replica: a072f007b74b41a2b1ed35bc6b456db9, State: Running, Role: LEADER
I20260927 22:47:14.807456 2666 consensus_queue.cc:237] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:14.810022 2138 catalog_manager.cc:5495] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a072f007b74b41a2b1ed35bc6b456db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:14.849239 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.844843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.0.97.2:43843)
0927 22:47:14.845016 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.845021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.845038 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:47:14.845096 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.845101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.845164 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.845232 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:14.845593 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.846331 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.847620 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.848086 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.848194 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.848576 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.848692 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.848761 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.848898 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.848968 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:47:14.861011 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.856416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43843 (local address 127.0.97.2:34393)
0927 22:47:14.856644 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:14.856661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:14.856783 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:14.857035 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:14.857039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:14.857070 (+ 31us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:14.857230 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.857237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.858259 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.858264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.859239 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:14.859249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.859997 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.860003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:14.860152 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.860183 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.860519 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.860528 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:47:14.860601 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:47:14.860728 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 22:47:14.861012 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:14.856483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34393 (local address 127.0.97.2:43843)
0927 22:47:14.856641 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:14.856647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:14.856724 (+ 77us) server_negotiation.cc:407] Connection header received
0927 22:47:14.856886 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:14.856891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:14.856950 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:14.857013 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:14.857334 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.858142 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.859369 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:14.859849 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:14.859968 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:14.860326 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:14.860431 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:14.860509 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:14.860630 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:47:14.860693 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:47:14.872186 2299 raft_consensus.cc:1261] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Refusing update from remote peer a072f007b74b41a2b1ed35bc6b456db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:14.872152 2573 raft_consensus.cc:1261] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Refusing update from remote peer a072f007b74b41a2b1ed35bc6b456db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:14.872735 2650 consensus_queue.cc:1035] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:14.873270 2650 consensus_queue.cc:1035] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:14.883882 2671 mvcc.cc:205] Tried to move back new op lower bound from 7334089666030936064 to 7334089665773498368. Current Snapshot: MvccSnapshot[applied={T|T < 7334089666030936064}]
I20260927 22:47:14.886427 2687 mvcc.cc:205] Tried to move back new op lower bound from 7334089666030936064 to 7334089665773498368. Current Snapshot: MvccSnapshot[applied={T|T < 7334089666030936064}]
W20260927 22:47:14.906518 2671 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:47:16.483870 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:16.484102 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:16.488113 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Sending DeleteTablet for 3 replicas of tablet 55a79710f7904a039af6e18522efb0c7
I20260927 22:47:16.488809 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a79710f7904a039af6e18522efb0c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42422
I20260927 22:47:16.488891 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a79710f7904a039af6e18522efb0c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36290
I20260927 22:47:16.488915 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a79710f7904a039af6e18522efb0c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43062
I20260927 22:47:16.501328 2723 tablet_replica.cc:330] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: stopping tablet replica
I20260927 22:47:16.501967 2723 raft_consensus.cc:2229] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:16.502841 2723 raft_consensus.cc:2258] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:16.504048 2725 tablet_replica.cc:330] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: stopping tablet replica
I20260927 22:47:16.504217 2725 raft_consensus.cc:2229] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:16.504673 2725 raft_consensus.cc:2258] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:16.508798 2725 ts_tablet_manager.cc:1905] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:16.512094 2723 ts_tablet_manager.cc:1905] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:16.514473 2725 ts_tablet_manager.cc:1918] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:47:16.514653 2725 log.cc:1196] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/55a79710f7904a039af6e18522efb0c7
I20260927 22:47:16.515182 2725 ts_tablet_manager.cc:1939] T 55a79710f7904a039af6e18522efb0c7 P a072f007b74b41a2b1ed35bc6b456db9: Deleting consensus metadata
I20260927 22:47:16.516427 2124 catalog_manager.cc:4841] TS a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843): tablet 55a79710f7904a039af6e18522efb0c7 (table test-table [id=8f731c7a183344bd89667c3638fc0a70]) successfully deleted
I20260927 22:47:16.517808 2723 ts_tablet_manager.cc:1918] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:47:16.517925 2723 log.cc:1196] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/55a79710f7904a039af6e18522efb0c7
I20260927 22:47:16.518260 2723 ts_tablet_manager.cc:1939] T 55a79710f7904a039af6e18522efb0c7 P 9880e9c3212448c996f38b6864b10f3d: Deleting consensus metadata
I20260927 22:47:16.519210 2125 catalog_manager.cc:4841] TS 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701): tablet 55a79710f7904a039af6e18522efb0c7 (table test-table [id=8f731c7a183344bd89667c3638fc0a70]) successfully deleted
I20260927 22:47:16.520419 2724 tablet_replica.cc:330] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: stopping tablet replica
I20260927 22:47:16.520608 2724 raft_consensus.cc:2229] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:16.520826 2724 raft_consensus.cc:2258] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:16.521703 2724 ts_tablet_manager.cc:1905] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:16.527107 2724 ts_tablet_manager.cc:1918] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:47:16.527213 2724 log.cc:1196] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/55a79710f7904a039af6e18522efb0c7
I20260927 22:47:16.527609 2724 ts_tablet_manager.cc:1939] T 55a79710f7904a039af6e18522efb0c7 P ff78a6ec81ea45a78e931519bdacda89: Deleting consensus metadata
I20260927 22:47:16.528492 2124 catalog_manager.cc:4841] TS ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223): tablet 55a79710f7904a039af6e18522efb0c7 (table test-table [id=8f731c7a183344bd89667c3638fc0a70]) successfully deleted
Sep 27 22:47:16 dist-test-slave-kr25 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549236, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:47:16 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549236, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:16.553858 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.541130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.0.97.62:46139)
0927 22:47:16.541768 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:16.541773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:16.541793 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:47:16.541843 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:16.541846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:16.541902 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:16.541993 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:16.543230 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.544024 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.545146 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.545539 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.545624 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.549504 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:16.549509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:16.549512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:16.549543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:16.551803 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:16.552333 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:16.552337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:16.552340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:16.552503 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:16.552762 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:16.552765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:16.552766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:16.553009 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:16.553184 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:16.553376 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 22:47:16.553551 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":381,"threads_started":1}
I20260927 22:47:16.557221 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:16.557780 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:16.563493 2275 tablet_service.cc:1467] Processing CreateTablet for tablet e64fdde5c2254512b6a065b21fd7615c (DEFAULT_TABLE table=test-table [id=35ea1866815d45f0874109aedac2d954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:16.563603 2413 tablet_service.cc:1467] Processing CreateTablet for tablet e64fdde5c2254512b6a065b21fd7615c (DEFAULT_TABLE table=test-table [id=35ea1866815d45f0874109aedac2d954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:16.563642 2553 tablet_service.cc:1467] Processing CreateTablet for tablet e64fdde5c2254512b6a065b21fd7615c (DEFAULT_TABLE table=test-table [id=35ea1866815d45f0874109aedac2d954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:16.563915 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64fdde5c2254512b6a065b21fd7615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:16.563979 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64fdde5c2254512b6a065b21fd7615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:16.564040 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64fdde5c2254512b6a065b21fd7615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:16.569518 2734 tablet_bootstrap.cc:492] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: Bootstrap starting.
I20260927 22:47:16.570801 2734 tablet_bootstrap.cc:654] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:16.572029 2736 tablet_bootstrap.cc:492] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: Bootstrap starting.
I20260927 22:47:16.572682 2735 tablet_bootstrap.cc:492] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: Bootstrap starting.
I20260927 22:47:16.573464 2736 tablet_bootstrap.cc:654] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:16.573645 2734 tablet_bootstrap.cc:492] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: No bootstrap required, opened a new log
I20260927 22:47:16.573802 2734 ts_tablet_manager.cc:1397] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:47:16.574313 2735 tablet_bootstrap.cc:654] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:16.574591 2734 raft_consensus.cc:348] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.574817 2734 raft_consensus.cc:374] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:16.574949 2734 raft_consensus.cc:729] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a072f007b74b41a2b1ed35bc6b456db9, State: Initialized, Role: FOLLOWER
I20260927 22:47:16.575232 2734 consensus_queue.cc:260] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.575802 2736 tablet_bootstrap.cc:492] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: No bootstrap required, opened a new log
I20260927 22:47:16.575945 2736 ts_tablet_manager.cc:1397] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:47:16.575975 2734 ts_tablet_manager.cc:1428] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:16.576673 2736 raft_consensus.cc:348] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.576875 2736 raft_consensus.cc:374] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:16.576957 2736 raft_consensus.cc:729] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9880e9c3212448c996f38b6864b10f3d, State: Initialized, Role: FOLLOWER
I20260927 22:47:16.577169 2736 consensus_queue.cc:260] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.577952 2736 ts_tablet_manager.cc:1428] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:16.579226 2735 tablet_bootstrap.cc:492] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: No bootstrap required, opened a new log
I20260927 22:47:16.579363 2735 ts_tablet_manager.cc:1397] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 22:47:16.580251 2735 raft_consensus.cc:348] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.580444 2735 raft_consensus.cc:374] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:16.580541 2735 raft_consensus.cc:729] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff78a6ec81ea45a78e931519bdacda89, State: Initialized, Role: FOLLOWER
I20260927 22:47:16.580780 2735 consensus_queue.cc:260] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.581526 2735 ts_tablet_manager.cc:1428] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:16.612637 2740 raft_consensus.cc:482] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:16.612838 2740 raft_consensus.cc:504] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.614365 2740 leader_election.cc:288] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:16.620872 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.614696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52559 (local address 127.0.97.2:43843)
0927 22:47:16.615178 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:16.615182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:16.616273 (+ 1091us) server_negotiation.cc:407] Connection header received
0927 22:47:16.616336 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:16.616340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:16.616394 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:16.616472 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:16.616951 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.617676 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.619275 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.619996 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.620117 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.620214 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:16.620451 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:47:16.620573 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":254,"threads_started":1}
I20260927 22:47:16.620873 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.614679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43843 (local address 127.0.97.1:52559)
0927 22:47:16.615965 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:16.615985 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:16.616153 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:16.616513 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:16.616517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:16.616542 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:16.616800 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.616808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.617809 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.617813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.619138 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.619149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.620171 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.620177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.620316 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.620339 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:47:16.620419 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:47:16.620542 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":471,"threads_started":1}
I20260927 22:47:16.621219 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46789 (local address 127.0.97.3:38223)
0927 22:47:16.616225 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:16.616230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:16.616445 (+ 215us) server_negotiation.cc:407] Connection header received
0927 22:47:16.616632 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:16.616636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:16.616688 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:16.616764 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:16.617288 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.618008 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.619592 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.620295 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.620400 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.620512 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:16.620786 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:47:16.620904 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":268,"threads_started":1}
I20260927 22:47:16.621251 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.615615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38223 (local address 127.0.97.1:46789)
0927 22:47:16.616350 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:16.616363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:16.616511 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:16.616805 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:16.616809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:16.616831 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:16.617144 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.617152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.618133 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.618137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.619428 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.619466 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.620496 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.620502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.620644 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.620666 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:47:16.620746 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:47:16.620854 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":221,"thread_start_us":289,"threads_started":1}
I20260927 22:47:16.621639 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64fdde5c2254512b6a065b21fd7615c" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a072f007b74b41a2b1ed35bc6b456db9" is_pre_election: true
I20260927 22:47:16.621870 2433 raft_consensus.cc:2454] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 0.
I20260927 22:47:16.621925 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64fdde5c2254512b6a065b21fd7615c" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89" is_pre_election: true
I20260927 22:47:16.622151 2573 raft_consensus.cc:2454] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 0.
I20260927 22:47:16.622380 2227 leader_election.cc:302] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, a072f007b74b41a2b1ed35bc6b456db9; no voters:
I20260927 22:47:16.622748 2740 raft_consensus.cc:2781] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:16.622846 2740 raft_consensus.cc:482] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:16.622934 2740 raft_consensus.cc:3037] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:16.625432 2740 raft_consensus.cc:504] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.625944 2740 leader_election.cc:288] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 election: Requested vote from peers a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:16.626430 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64fdde5c2254512b6a065b21fd7615c" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89"
I20260927 22:47:16.626437 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64fdde5c2254512b6a065b21fd7615c" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a072f007b74b41a2b1ed35bc6b456db9"
I20260927 22:47:16.626633 2573 raft_consensus.cc:3037] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:16.626637 2433 raft_consensus.cc:3037] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:16.628569 2573 raft_consensus.cc:2454] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 1.
I20260927 22:47:16.628644 2433 raft_consensus.cc:2454] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 1.
I20260927 22:47:16.628887 2227 leader_election.cc:302] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, ff78a6ec81ea45a78e931519bdacda89; no voters:
I20260927 22:47:16.629128 2740 raft_consensus.cc:2781] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:16.629483 2740 raft_consensus.cc:686] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 1 LEADER]: Becoming Leader. State: Replica: 9880e9c3212448c996f38b6864b10f3d, State: Running, Role: LEADER
I20260927 22:47:16.629837 2740 consensus_queue.cc:237] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.632695 2137 catalog_manager.cc:5495] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9880e9c3212448c996f38b6864b10f3d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9880e9c3212448c996f38b6864b10f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:16.658701 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.653210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.0.97.1:40701)
0927 22:47:16.653708 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:16.653712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:16.653730 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:16.653805 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:16.653808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:16.653879 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:16.654010 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:16.654317 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.655069 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.656769 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.657157 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.657241 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.657611 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:16.658156 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:16.658257 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:16.658374 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 22:47:16.658435 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":292,"threads_started":1}
I20260927 22:47:16.673578 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.668519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34687 (local address 127.0.97.2:43843)
0927 22:47:16.668705 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:16.668710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:16.669041 (+ 331us) server_negotiation.cc:407] Connection header received
0927 22:47:16.669211 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:16.669217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:16.669283 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:16.669354 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:16.669831 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.670575 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.672037 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.672424 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.672525 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.672929 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:16.673032 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:16.673116 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:16.673227 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 22:47:16.673300 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:47:16.673664 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.668348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43843 (local address 127.0.97.1:34687)
0927 22:47:16.668952 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:16.668965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:16.669089 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:16.669376 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:16.669381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:16.669391 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:16.669693 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.669702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.670695 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.670700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.671912 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.671925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.672570 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.672576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.672713 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.672761 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:16.673120 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:16.673130 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:47:16.673202 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 22:47:16.673330 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":310,"threads_started":1}
I20260927 22:47:16.681962 2573 raft_consensus.cc:1261] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Refusing update from remote peer 9880e9c3212448c996f38b6864b10f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:16.681994 2433 raft_consensus.cc:1261] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Refusing update from remote peer 9880e9c3212448c996f38b6864b10f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:16.682600 2746 consensus_queue.cc:1035] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:16.682907 2740 consensus_queue.cc:1035] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:16.706420 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.700674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40865 (local address 127.0.97.1:40701)
0927 22:47:16.700816 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:16.700820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:16.701738 (+ 918us) server_negotiation.cc:407] Connection header received
0927 22:47:16.701905 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:16.701910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:16.701968 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:16.702032 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:16.702454 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.703349 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.704857 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.705293 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.705396 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.705771 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:16.705861 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:16.705928 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:16.706085 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:47:16.706150 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:47:16.707089 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:16.701128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40701 (local address 127.0.97.2:40865)
0927 22:47:16.701647 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:16.701662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:16.701780 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:16.702055 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:16.702059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:16.702071 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:16.702317 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.702325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.703475 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.703480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.704734 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:16.704747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:16.705444 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:16.705451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:16.705597 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:16.705632 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:16.705945 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:16.705956 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:47:16.706022 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 22:47:16.706146 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260927 22:47:16.735466 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:16.735735 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:16.738569 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Sending DeleteTablet for 3 replicas of tablet e64fdde5c2254512b6a065b21fd7615c
I20260927 22:47:16.739238 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet e64fdde5c2254512b6a065b21fd7615c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42422
I20260927 22:47:16.739416 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet e64fdde5c2254512b6a065b21fd7615c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36290
I20260927 22:47:16.739739 2723 tablet_replica.cc:330] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: stopping tablet replica
I20260927 22:47:16.739924 2723 raft_consensus.cc:2229] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:16.740176 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet e64fdde5c2254512b6a065b21fd7615c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43062
I20260927 22:47:16.740370 2723 raft_consensus.cc:2258] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:16.740448 2725 tablet_replica.cc:330] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: stopping tablet replica
I20260927 22:47:16.740653 2724 tablet_replica.cc:330] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: stopping tablet replica
I20260927 22:47:16.740636 2725 raft_consensus.cc:2229] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:16.740597 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:47:16.740831 2725 raft_consensus.cc:2258] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:16.740841 2724 raft_consensus.cc:2229] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:16.741003 2724 raft_consensus.cc:2258] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 22:47:16.741168 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:16.741894 2725 ts_tablet_manager.cc:1905] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:16.742643 2724 ts_tablet_manager.cc:1905] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:16.744860 2723 ts_tablet_manager.cc:1905] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:16.748479 2725 ts_tablet_manager.cc:1918] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:16.748656 2725 log.cc:1196] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/e64fdde5c2254512b6a065b21fd7615c
I20260927 22:47:16.749074 2725 ts_tablet_manager.cc:1939] T e64fdde5c2254512b6a065b21fd7615c P a072f007b74b41a2b1ed35bc6b456db9: Deleting consensus metadata
I20260927 22:47:16.749187 2724 ts_tablet_manager.cc:1918] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:16.749365 2724 log.cc:1196] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/e64fdde5c2254512b6a065b21fd7615c
I20260927 22:47:16.749765 2724 ts_tablet_manager.cc:1939] T e64fdde5c2254512b6a065b21fd7615c P ff78a6ec81ea45a78e931519bdacda89: Deleting consensus metadata
I20260927 22:47:16.750125 2124 catalog_manager.cc:4841] TS a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843): tablet e64fdde5c2254512b6a065b21fd7615c (table test-table [id=35ea1866815d45f0874109aedac2d954]) successfully deleted
I20260927 22:47:16.750665 2124 catalog_manager.cc:4841] TS ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223): tablet e64fdde5c2254512b6a065b21fd7615c (table test-table [id=35ea1866815d45f0874109aedac2d954]) successfully deleted
I20260927 22:47:16.751814 2723 ts_tablet_manager.cc:1918] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:16.752017 2723 log.cc:1196] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/e64fdde5c2254512b6a065b21fd7615c
I20260927 22:47:16.752722 2723 ts_tablet_manager.cc:1939] T e64fdde5c2254512b6a065b21fd7615c P 9880e9c3212448c996f38b6864b10f3d: Deleting consensus metadata
I20260927 22:47:16.753873 2125 catalog_manager.cc:4841] TS 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701): tablet e64fdde5c2254512b6a065b21fd7615c (table test-table [id=35ea1866815d45f0874109aedac2d954]) successfully deleted
I20260927 22:47:16.755577 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 80355f52d8d1470ca0e08ea889cb4420 (DEFAULT_TABLE table=test-table [id=c00d0728135b4fad92015ebb3f2133c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:16.755891 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 80355f52d8d1470ca0e08ea889cb4420 (DEFAULT_TABLE table=test-table [id=c00d0728135b4fad92015ebb3f2133c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:16.755981 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80355f52d8d1470ca0e08ea889cb4420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:16.756263 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80355f52d8d1470ca0e08ea889cb4420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:16.760823 2735 tablet_bootstrap.cc:492] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: Bootstrap starting.
I20260927 22:47:16.761328 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 80355f52d8d1470ca0e08ea889cb4420 (DEFAULT_TABLE table=test-table [id=c00d0728135b4fad92015ebb3f2133c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:16.761714 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80355f52d8d1470ca0e08ea889cb4420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:16.762306 2735 tablet_bootstrap.cc:654] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:16.762823 2736 tablet_bootstrap.cc:492] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: Bootstrap starting.
I20260927 22:47:16.764369 2736 tablet_bootstrap.cc:654] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:16.767871 2734 tablet_bootstrap.cc:492] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: Bootstrap starting.
I20260927 22:47:16.769111 2734 tablet_bootstrap.cc:654] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:16.770931 2735 tablet_bootstrap.cc:492] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: No bootstrap required, opened a new log
I20260927 22:47:16.771100 2735 ts_tablet_manager.cc:1397] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:47:16.771121 2734 tablet_bootstrap.cc:492] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: No bootstrap required, opened a new log
I20260927 22:47:16.771260 2734 ts_tablet_manager.cc:1397] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:16.771864 2734 raft_consensus.cc:348] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.771871 2735 raft_consensus.cc:348] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.772045 2734 raft_consensus.cc:374] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:16.772125 2734 raft_consensus.cc:729] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a072f007b74b41a2b1ed35bc6b456db9, State: Initialized, Role: FOLLOWER
I20260927 22:47:16.772187 2735 raft_consensus.cc:374] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:16.772336 2735 raft_consensus.cc:729] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff78a6ec81ea45a78e931519bdacda89, State: Initialized, Role: FOLLOWER
I20260927 22:47:16.772354 2734 consensus_queue.cc:260] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.772601 2735 consensus_queue.cc:260] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.772769 2734 ts_tablet_manager.cc:1428] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 22:47:16.773129 2735 ts_tablet_manager.cc:1428] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:16.774857 2736 tablet_bootstrap.cc:492] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: No bootstrap required, opened a new log
I20260927 22:47:16.775085 2736 ts_tablet_manager.cc:1397] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260927 22:47:16.776430 2736 raft_consensus.cc:348] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.776744 2736 raft_consensus.cc:374] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:16.776829 2736 raft_consensus.cc:729] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9880e9c3212448c996f38b6864b10f3d, State: Initialized, Role: FOLLOWER
I20260927 22:47:16.777038 2736 consensus_queue.cc:260] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.777561 2736 ts_tablet_manager.cc:1428] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:16.859757 2740 raft_consensus.cc:482] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:16.859934 2740 raft_consensus.cc:504] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.860549 2740 leader_election.cc:288] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:16.861119 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80355f52d8d1470ca0e08ea889cb4420" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a072f007b74b41a2b1ed35bc6b456db9" is_pre_election: true
I20260927 22:47:16.861160 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80355f52d8d1470ca0e08ea889cb4420" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89" is_pre_election: true
I20260927 22:47:16.861328 2433 raft_consensus.cc:2454] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 0.
I20260927 22:47:16.861469 2573 raft_consensus.cc:2454] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 0.
I20260927 22:47:16.861869 2227 leader_election.cc:302] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, a072f007b74b41a2b1ed35bc6b456db9; no voters:
I20260927 22:47:16.862365 2740 raft_consensus.cc:2781] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:16.862480 2740 raft_consensus.cc:482] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:16.862591 2740 raft_consensus.cc:3037] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:16.865672 2740 raft_consensus.cc:504] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.866295 2740 leader_election.cc:288] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 election: Requested vote from peers a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843), ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223)
I20260927 22:47:16.866715 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80355f52d8d1470ca0e08ea889cb4420" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff78a6ec81ea45a78e931519bdacda89"
I20260927 22:47:16.866757 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80355f52d8d1470ca0e08ea889cb4420" candidate_uuid: "9880e9c3212448c996f38b6864b10f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a072f007b74b41a2b1ed35bc6b456db9"
I20260927 22:47:16.866910 2573 raft_consensus.cc:3037] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:16.866958 2433 raft_consensus.cc:3037] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:16.868822 2573 raft_consensus.cc:2454] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 1.
I20260927 22:47:16.869186 2227 leader_election.cc:302] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9880e9c3212448c996f38b6864b10f3d, ff78a6ec81ea45a78e931519bdacda89; no voters:
I20260927 22:47:16.869432 2740 raft_consensus.cc:2781] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:16.869552 2740 raft_consensus.cc:686] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 1 LEADER]: Becoming Leader. State: Replica: 9880e9c3212448c996f38b6864b10f3d, State: Running, Role: LEADER
I20260927 22:47:16.869796 2433 raft_consensus.cc:2454] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9880e9c3212448c996f38b6864b10f3d in term 1.
I20260927 22:47:16.869829 2740 consensus_queue.cc:237] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } }
I20260927 22:47:16.873283 2137 catalog_manager.cc:5495] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9880e9c3212448c996f38b6864b10f3d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9880e9c3212448c996f38b6864b10f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9880e9c3212448c996f38b6864b10f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:16.904414 2573 raft_consensus.cc:1261] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Refusing update from remote peer 9880e9c3212448c996f38b6864b10f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:16.904451 2433 raft_consensus.cc:1261] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Refusing update from remote peer 9880e9c3212448c996f38b6864b10f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:16.904861 2740 consensus_queue.cc:1035] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff78a6ec81ea45a78e931519bdacda89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:16.905133 2746 consensus_queue.cc:1035] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a072f007b74b41a2b1ed35bc6b456db9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:47:16.932286 2708 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:47:18.513130 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:18.513329 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:18.516957 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d060e8516a249eda6e81288fc4db556: Sending DeleteTablet for 3 replicas of tablet 80355f52d8d1470ca0e08ea889cb4420
I20260927 22:47:18.517594 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 80355f52d8d1470ca0e08ea889cb4420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42422
I20260927 22:47:18.517751 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 80355f52d8d1470ca0e08ea889cb4420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43062
I20260927 22:47:18.517849 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 80355f52d8d1470ca0e08ea889cb4420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36290
I20260927 22:47:18.518468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2210
I20260927 22:47:18.531010 2791 tablet_replica.cc:330] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: stopping tablet replica
I20260927 22:47:18.531447 2791 raft_consensus.cc:2229] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:18.531823 2791 raft_consensus.cc:2258] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:18.533177 2789 tablet_replica.cc:330] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: stopping tablet replica
I20260927 22:47:18.533397 2789 raft_consensus.cc:2229] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:18.533783 2789 raft_consensus.cc:2258] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:18.534369 2790 tablet_replica.cc:330] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: stopping tablet replica
I20260927 22:47:18.534631 2791 ts_tablet_manager.cc:1905] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:18.535106 2790 raft_consensus.cc:2229] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:18.535158 2789 ts_tablet_manager.cc:1905] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:18.535385 2790 raft_consensus.cc:2258] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:18.536772 2790 ts_tablet_manager.cc:1905] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:18.540686 2789 ts_tablet_manager.cc:1918] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:47:18.540804 2789 log.cc:1196] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/80355f52d8d1470ca0e08ea889cb4420
Sep 27 22:47:18 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549236, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:18.541178 2791 ts_tablet_manager.cc:1918] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:47:18.541234 2789 ts_tablet_manager.cc:1939] T 80355f52d8d1470ca0e08ea889cb4420 P 9880e9c3212448c996f38b6864b10f3d: Deleting consensus metadata
I20260927 22:47:18.541316 2791 log.cc:1196] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/80355f52d8d1470ca0e08ea889cb4420
I20260927 22:47:18.541765 2791 ts_tablet_manager.cc:1939] T 80355f52d8d1470ca0e08ea889cb4420 P ff78a6ec81ea45a78e931519bdacda89: Deleting consensus metadata
I20260927 22:47:18.542268 2125 catalog_manager.cc:4841] TS 9880e9c3212448c996f38b6864b10f3d (127.0.97.1:40701): tablet 80355f52d8d1470ca0e08ea889cb4420 (table test-table [id=c00d0728135b4fad92015ebb3f2133c9]) successfully deleted
I20260927 22:47:18.542806 2124 catalog_manager.cc:4841] TS ff78a6ec81ea45a78e931519bdacda89 (127.0.97.3:38223): tablet 80355f52d8d1470ca0e08ea889cb4420 (table test-table [id=c00d0728135b4fad92015ebb3f2133c9]) successfully deleted
I20260927 22:47:18.543121 2790 ts_tablet_manager.cc:1918] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:47:18.543247 2790 log.cc:1196] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/80355f52d8d1470ca0e08ea889cb4420
I20260927 22:47:18.543709 2790 ts_tablet_manager.cc:1939] T 80355f52d8d1470ca0e08ea889cb4420 P a072f007b74b41a2b1ed35bc6b456db9: Deleting consensus metadata
I20260927 22:47:18.544683 2124 catalog_manager.cc:4841] TS a072f007b74b41a2b1ed35bc6b456db9 (127.0.97.2:43843): tablet 80355f52d8d1470ca0e08ea889cb4420 (table test-table [id=c00d0728135b4fad92015ebb3f2133c9]) successfully deleted
I20260927 22:47:18.548112 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:18.520234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.0.97.1:40701)
0927 22:47:18.526786 (+ 6552us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:18.526790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:18.526814 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:47:18.526883 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:18.526887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:18.526954 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:18.527070 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:18.528960 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:18.530255 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:18.532231 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:18.532887 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:18.533034 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:18.542657 (+ 9623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:18.542675 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:18.542696 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:18.542733 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:18.545631 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:18.546220 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:18.546230 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:18.546233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:18.546353 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:18.546697 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:18.546702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:18.546706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:18.547031 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:18.547275 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:18.547610 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:47:18.547812 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6434,"thread_start_us":6328,"threads_started":1}
W20260927 22:47:18.548677 2340 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42434
W20260927 22:47:18.549247 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:18.549432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 2210
I20260927 22:47:18.567020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2349
Sep 27 22:47:18 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549236, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:18.580387 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:18.567689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.0.97.2:43843)
0927 22:47:18.568129 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:18.568133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:18.568149 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:47:18.568199 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:18.568203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:18.568259 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:18.568349 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:18.569438 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:18.570177 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:18.571397 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:18.571811 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:18.571910 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:18.575592 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:18.575606 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:18.575621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:18.575658 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:18.578248 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:18.578776 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:18.578782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:18.578784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:18.578920 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:18.579192 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:18.579195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:18.579196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:18.579562 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:18.579782 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:18.579970 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 22:47:18.580122 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
W20260927 22:47:18.580984 2479 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43090
W20260927 22:47:18.581538 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2349: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:18.581660 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 2349
I20260927 22:47:18.601569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2488
Sep 27 22:47:18 dist-test-slave-kr25 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549236, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:18.614804 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:18.602221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.0.97.3:38223)
0927 22:47:18.602675 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:18.602679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:18.602694 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:47:18.602744 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:18.602747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:18.602810 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:18.602894 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:18.604112 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:18.604823 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:18.605942 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:18.606310 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:18.606396 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:18.610145 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:18.610158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:18.610175 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:18.610203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:18.612653 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:18.613225 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:18.613232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:18.613234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:18.613360 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:18.613630 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:18.613633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:18.613635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:18.613949 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:18.614146 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:18.614340 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:47:18.614483 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
W20260927 22:47:18.615360 2618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36304
W20260927 22:47:18.615921 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2488: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:18.616063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 2488
I20260927 22:47:18.632741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2107
I20260927 22:47:18.633769 2179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:18.745581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 2107
2026-09-27T22:47:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7885 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:18 dist-test-slave-kr25 krb5kdc[2799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:18 dist-test-slave-kr25 krb5kdc[2799](info): set up 2 sockets
Sep 27 22:47:18 dist-test-slave-kr25 krb5kdc[2799](info): commencing operation
krb5kdc: starting...
W20260927 22:47:19.727669 388 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:19 dist-test-slave-kr25 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:19.888765 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35751
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:20.187750 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:20.188031 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:20.188159 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:20.195528 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:20.195624 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:20.195693 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:20.195767 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:20.195822 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:20.205286 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:20.206532 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:20.207777 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:20.218791 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:20.218703 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:20.218739 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:20.219033 2812 server_base.cc:1034] running on GCE node
I20260927 22:47:20.220054 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:20.221377 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:20.222561 2812 hybrid_clock.cc:648] HybridClock initialized: now 1790549240222505 us; error 44 us; skew 500 ppm
I20260927 22:47:20.222860 2812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:20 dist-test-slave-kr25 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:20.225652 2812 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:20.227032 2812 webserver.cc:460] Webserver started at http://127.0.97.62:45833/ using document root <none> and password file <none>
I20260927 22:47:20.227679 2812 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:20.227774 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:20.228024 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:20.230690 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "e2bea49bc43f4c30917ab6bd342fb8d5"
format_stamp: "Formatted at 2026-09-27 22:47:20 on dist-test-slave-kr25"
server_key: "bf318dada91a836c3127f3b1fd562b86"
server_key_iv: "7182ecf8774d12bd9bbf132de2e41012"
server_key_version: "encryptionkey@0"
I20260927 22:47:20.231390 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "e2bea49bc43f4c30917ab6bd342fb8d5"
format_stamp: "Formatted at 2026-09-27 22:47:20 on dist-test-slave-kr25"
server_key: "bf318dada91a836c3127f3b1fd562b86"
server_key_iv: "7182ecf8774d12bd9bbf132de2e41012"
server_key_version: "encryptionkey@0"
I20260927 22:47:20.236061 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:47:20.239099 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:20.240321 2812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:20.240466 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "e2bea49bc43f4c30917ab6bd342fb8d5"
format_stamp: "Formatted at 2026-09-27 22:47:20 on dist-test-slave-kr25"
server_key: "bf318dada91a836c3127f3b1fd562b86"
server_key_iv: "7182ecf8774d12bd9bbf132de2e41012"
server_key_version: "encryptionkey@0"
I20260927 22:47:20.240631 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:20.252877 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:20.256454 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:20.256690 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:20.272548 2812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35751
I20260927 22:47:20.272576 2878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35751 every 8 connection(s)
I20260927 22:47:20.273860 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:20.277288 2879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:20.283943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 2812
I20260927 22:47:20.284134 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:20.284701 388 external_mini_cluster.cc:1428] Setting key 951ba7878330a9461b0dd99bd77c01ac
I20260927 22:47:20.286979 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5: Bootstrap starting.
I20260927 22:47:20.290046 2879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:20.291265 2879 log.cc:824] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:20.294157 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5: No bootstrap required, opened a new log
Sep 27 22:47:20 dist-test-slave-kr25 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:20.302783 2879 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35751 } }
I20260927 22:47:20.303228 2879 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:20.303323 2879 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2bea49bc43f4c30917ab6bd342fb8d5, State: Initialized, Role: FOLLOWER
I20260927 22:47:20.303987 2879 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35751 } }
I20260927 22:47:20.304221 2879 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:20.304344 2879 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:20.304508 2879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:20.306432 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:20.286496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.0.97.62:35751)
0927 22:47:20.287191 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:20.287203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:20.287241 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:47:20.288876 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:20.288896 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:20.289251 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:20.289540 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:20.291263 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:20.292857 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:20.294495 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:20.295124 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:20.295244 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:20.299632 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:20.299640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:20.299653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:20.299691 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:20.302528 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:20.303262 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:20.303270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:20.303278 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:20.303501 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:20.303868 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:20.303871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:20.303873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:20.304433 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:20.304712 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:20.305089 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 22:47:20.305419 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":322,"threads_started":1}
I20260927 22:47:20.307026 2879 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35751 } }
I20260927 22:47:20.307569 2879 leader_election.cc:302] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2bea49bc43f4c30917ab6bd342fb8d5; no voters:
I20260927 22:47:20.308207 2879 leader_election.cc:288] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:20.308283 2884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:20.310155 2879 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:20.315140 2884 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [term 1 LEADER]: Becoming Leader. State: Replica: e2bea49bc43f4c30917ab6bd342fb8d5, State: Running, Role: LEADER
I20260927 22:47:20.315640 2884 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35751 } }
I20260927 22:47:20.318435 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35751 } } }
I20260927 22:47:20.318574 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2bea49bc43f4c30917ab6bd342fb8d5. Latest consensus state: current_term: 1 leader_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2bea49bc43f4c30917ab6bd342fb8d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35751 } } }
I20260927 22:47:20.319684 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:20.319685 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:20.321033 2898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:20.325069 2898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:20.332034 2898 catalog_manager.cc:1347] Generated new cluster ID: 5c1d44ae9623495dada3b8a4e0e71d7c
I20260927 22:47:20.332144 2898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:20.354104 2898 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:20.355348 2898 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:20.360539 2898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2bea49bc43f4c30917ab6bd342fb8d5: Generated new TSK 0
I20260927 22:47:20.361292 2898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:20.400920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35751
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:20.706976 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:20.707232 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:20.707360 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:20.714239 2905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:20.714335 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:20.714499 2905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:20.723671 2905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:20.724865 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:20.726115 2905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:20.737897 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:20.737960 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:20.738528 2905 server_base.cc:1034] running on GCE node
W20260927 22:47:20.738546 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:20.739116 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:20.739976 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:20.741187 2905 hybrid_clock.cc:648] HybridClock initialized: now 1790549240741133 us; error 44 us; skew 500 ppm
I20260927 22:47:20.741488 2905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:20 dist-test-slave-kr25 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:20.744303 2905 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:20.745673 2905 webserver.cc:460] Webserver started at http://127.0.97.1:43001/ using document root <none> and password file <none>
I20260927 22:47:20.746261 2905 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:20.746364 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:20.746642 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:20.749361 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "82c6685f68f14d2296fa0aa5cb1b1d9c"
format_stamp: "Formatted at 2026-09-27 22:47:20 on dist-test-slave-kr25"
server_key: "d50ef6d37561715c5324c4bdacd9b416"
server_key_iv: "c9ffc7c521b53e8a9bca2f4d48d12141"
server_key_version: "encryptionkey@0"
I20260927 22:47:20.750080 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "82c6685f68f14d2296fa0aa5cb1b1d9c"
format_stamp: "Formatted at 2026-09-27 22:47:20 on dist-test-slave-kr25"
server_key: "d50ef6d37561715c5324c4bdacd9b416"
server_key_iv: "c9ffc7c521b53e8a9bca2f4d48d12141"
server_key_version: "encryptionkey@0"
I20260927 22:47:20.754884 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:47:20.758167 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:20.759377 2905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:47:20.759581 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "82c6685f68f14d2296fa0aa5cb1b1d9c"
format_stamp: "Formatted at 2026-09-27 22:47:20 on dist-test-slave-kr25"
server_key: "d50ef6d37561715c5324c4bdacd9b416"
server_key_iv: "c9ffc7c521b53e8a9bca2f4d48d12141"
server_key_version: "encryptionkey@0"
I20260927 22:47:20.759758 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:20.772315 2905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:20.775784 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:20.776032 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:20.776850 2905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:20.778047 2905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:20.778119 2905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:20.778228 2905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:20.778278 2905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:20.806480 2905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33811
I20260927 22:47:20.806486 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33811 every 8 connection(s)
I20260927 22:47:20.807773 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:20.816129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 2905
I20260927 22:47:20.816314 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:20.816879 388 external_mini_cluster.cc:1428] Setting key ff24dcf95f4b5b76790eee9786f39e3c
Sep 27 22:47:20 dist-test-slave-kr25 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:20.826133 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:20.810068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47191 (local address 127.0.97.62:35751)
0927 22:47:20.810473 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:20.810477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:20.811307 (+ 830us) server_negotiation.cc:407] Connection header received
0927 22:47:20.813037 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:20.813042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:20.813137 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:20.813231 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:20.814647 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:20.815392 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:20.817057 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:20.817494 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:20.817622 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:20.820934 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:20.820939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:20.820942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:20.820971 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:20.823275 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:20.824115 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:20.824119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:20.824121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:20.824234 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:20.824639 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:20.824642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:20.824644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:20.824871 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:20.824985 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:20.825727 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 22:47:20.825870 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":202,"threads_started":1}
I20260927 22:47:20.827411 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:20.810532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35751 (local address 127.0.97.1:47191)
0927 22:47:20.811169 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:20.811205 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:20.812790 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:20.813464 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:20.813471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:20.813949 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:20.814472 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:20.814489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:20.815542 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:20.815546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:20.816898 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:20.816914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:20.817733 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:20.817740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:20.818015 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:20.818701 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:20.818729 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:20.820730 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:20.823400 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:20.823408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:20.823424 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:20.824015 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:20.824324 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:20.824328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:20.824332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:20.824548 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:20.824973 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:20.824981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:20.825362 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 22:47:20.825678 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 22:47:20.826148 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":235,"threads_started":1}
I20260927 22:47:20.828923 3033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35751
I20260927 22:47:20.829232 3033 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:20.830060 3033 heartbeater.cc:511] Master 127.0.97.62:35751 requested a full tablet report, sending...
I20260927 22:47:20.832376 2844 ts_manager.cc:194] Registered new tserver with Master: 82c6685f68f14d2296fa0aa5cb1b1d9c (127.0.97.1:33811)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:20.834333 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:20.855906 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35751
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:21.157473 3039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:21.157759 3039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:21.157872 3039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:21.165073 3039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:21.165194 3039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:21.165339 3039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:21.174906 3039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:21.176184 3039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:21.177402 3039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:21.189235 3044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:21.189390 3045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:21.189425 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:21.190255 3039 server_base.cc:1034] running on GCE node
I20260927 22:47:21.190752 3039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:21.191581 3039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:21.192767 3039 hybrid_clock.cc:648] HybridClock initialized: now 1790549241192714 us; error 49 us; skew 500 ppm
I20260927 22:47:21.193013 3039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:21 dist-test-slave-kr25 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:21.195873 3039 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:21.197124 3039 webserver.cc:460] Webserver started at http://127.0.97.2:34391/ using document root <none> and password file <none>
I20260927 22:47:21.197718 3039 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:21.197810 3039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:21.198046 3039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:21.200765 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "5a91b16b42bc49c7bfcade82b3e53dbc"
format_stamp: "Formatted at 2026-09-27 22:47:21 on dist-test-slave-kr25"
server_key: "2aaffe5b5c15fc2dbd22cd573ab4648d"
server_key_iv: "fdbdf9db809e35edfac4bf49ef476d57"
server_key_version: "encryptionkey@0"
I20260927 22:47:21.201440 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5a91b16b42bc49c7bfcade82b3e53dbc"
format_stamp: "Formatted at 2026-09-27 22:47:21 on dist-test-slave-kr25"
server_key: "2aaffe5b5c15fc2dbd22cd573ab4648d"
server_key_iv: "fdbdf9db809e35edfac4bf49ef476d57"
server_key_version: "encryptionkey@0"
I20260927 22:47:21.206248 3039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:47:21.209522 3054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:21.210886 3039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:21.211028 3039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "5a91b16b42bc49c7bfcade82b3e53dbc"
format_stamp: "Formatted at 2026-09-27 22:47:21 on dist-test-slave-kr25"
server_key: "2aaffe5b5c15fc2dbd22cd573ab4648d"
server_key_iv: "fdbdf9db809e35edfac4bf49ef476d57"
server_key_version: "encryptionkey@0"
I20260927 22:47:21.211216 3039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:21.236758 3039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:21.240484 3039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:21.240738 3039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:21.241801 3039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:21.243481 3039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:21.243568 3039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 22:47:21.243677 3039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:21.243738 3039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:21.272900 3039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40331
I20260927 22:47:21.272939 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40331 every 8 connection(s)
I20260927 22:47:21.274084 3039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:21.280740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3039
I20260927 22:47:21.280918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:21.281385 388 external_mini_cluster.cc:1428] Setting key 0085d471763fd6079708e77d109e4ea7
Sep 27 22:47:21 dist-test-slave-kr25 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:21.294762 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:21.276583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54241 (local address 127.0.97.62:35751)
0927 22:47:21.276738 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:21.276742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:21.278036 (+ 1294us) server_negotiation.cc:407] Connection header received
0927 22:47:21.279806 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:21.279811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:21.279892 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:21.279976 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:21.281441 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.282309 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.284534 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.285181 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.285291 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:21.289274 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:21.289279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:21.289282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:21.289313 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:21.291700 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:21.292603 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:21.292607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:21.292610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:21.292755 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:21.293170 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:21.293174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:21.293176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:21.293438 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:21.293564 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:21.294219 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 22:47:21.294448 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:47:21.295804 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:21.277146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35751 (local address 127.0.97.2:54241)
0927 22:47:21.277875 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:21.277915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:21.279567 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:21.280209 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:21.280218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:21.280686 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:21.281289 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:21.281307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.282464 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.282470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:21.284383 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:21.284398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.285394 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.285401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:21.285682 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:21.286415 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:21.286443 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:21.289086 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:21.291826 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:21.291835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:21.291853 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:21.292467 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:21.292860 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:21.292865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:21.292869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:21.293051 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:21.293563 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:21.293572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:21.293882 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 22:47:21.294114 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:47:21.294638 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":277,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:47:21.297223 3167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35751
I20260927 22:47:21.297499 3167 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:21.298152 3167 heartbeater.cc:511] Master 127.0.97.62:35751 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:47:21.299405 2844 ts_manager.cc:194] Registered new tserver with Master: 5a91b16b42bc49c7bfcade82b3e53dbc (127.0.97.2:40331)
I20260927 22:47:21.300225 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:21.318728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35751
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:21.620879 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:21.621133 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:21.621259 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:21.628193 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:21.628291 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:21.628435 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:21.638105 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:21.639276 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:21.640527 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:21.652277 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:21.652354 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:21.652307 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:21.653244 3172 server_base.cc:1034] running on GCE node
I20260927 22:47:21.653776 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:21.654585 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:21.655774 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790549241655729 us; error 39 us; skew 500 ppm
I20260927 22:47:21.656055 3172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:21 dist-test-slave-kr25 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:21.658826 3172 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:21.660120 3172 webserver.cc:460] Webserver started at http://127.0.97.3:46549/ using document root <none> and password file <none>
I20260927 22:47:21.660722 3172 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:21.660830 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:21.661095 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:21.663956 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "873d318a8610495e83f5e3dfe94eedc9"
format_stamp: "Formatted at 2026-09-27 22:47:21 on dist-test-slave-kr25"
server_key: "66332e186ca9a8df59e6715b1a96f47a"
server_key_iv: "1a4338985257c5886c1f40237d540046"
server_key_version: "encryptionkey@0"
I20260927 22:47:21.664680 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "873d318a8610495e83f5e3dfe94eedc9"
format_stamp: "Formatted at 2026-09-27 22:47:21 on dist-test-slave-kr25"
server_key: "66332e186ca9a8df59e6715b1a96f47a"
server_key_iv: "1a4338985257c5886c1f40237d540046"
server_key_version: "encryptionkey@0"
I20260927 22:47:21.669340 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:21.672500 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:21.673759 3172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:21.673898 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "873d318a8610495e83f5e3dfe94eedc9"
format_stamp: "Formatted at 2026-09-27 22:47:21 on dist-test-slave-kr25"
server_key: "66332e186ca9a8df59e6715b1a96f47a"
server_key_iv: "1a4338985257c5886c1f40237d540046"
server_key_version: "encryptionkey@0"
I20260927 22:47:21.674072 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:21.701584 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:21.705170 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:21.705413 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:21.706264 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:21.707499 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:21.707572 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:21.707675 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:21.707726 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:21.735797 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33145
I20260927 22:47:21.735843 3299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33145 every 8 connection(s)
I20260927 22:47:21.736963 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:21.743958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3172
I20260927 22:47:21.744122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:21.744634 388 external_mini_cluster.cc:1428] Setting key 4c190432468382f573cc5b7130bcde50
Sep 27 22:47:21 dist-test-slave-kr25 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:21.755128 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:21.739201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50085 (local address 127.0.97.62:35751)
0927 22:47:21.739402 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:21.739408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:21.740531 (+ 1123us) server_negotiation.cc:407] Connection header received
0927 22:47:21.742201 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:21.742207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:21.742277 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:21.742366 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:21.743784 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.744544 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.746948 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.747403 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.747531 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:21.750517 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:21.750521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:21.750524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:21.750551 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:21.752775 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:21.753401 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:21.753405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:21.753407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:21.753529 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:21.753839 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:21.753842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:21.753844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:21.754075 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:21.754181 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:21.754734 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 22:47:21.754881 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:47:21.756132 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:21.739741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35751 (local address 127.0.97.3:50085)
0927 22:47:21.740376 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:21.740412 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:21.741956 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:21.742569 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:21.742576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:21.743020 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:21.743624 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:21.743648 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.744660 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.744665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:21.746403 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:21.746416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:21.747603 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:21.747608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:21.747791 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:21.748382 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:21.748404 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:21.750352 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:21.752889 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:21.752903 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:21.752914 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:21.753318 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:21.753602 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:21.753605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:21.753607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:21.753764 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:21.754166 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:21.754171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:21.754449 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 22:47:21.754666 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 22:47:21.755019 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":244,"threads_started":1}
I20260927 22:47:21.757314 3300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35751
I20260927 22:47:21.757642 3300 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:21.758315 3300 heartbeater.cc:511] Master 127.0.97.62:35751 requested a full tablet report, sending...
I20260927 22:47:21.759685 2844 ts_manager.cc:194] Registered new tserver with Master: 873d318a8610495e83f5e3dfe94eedc9 (127.0.97.3:33145)
I20260927 22:47:21.760452 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50085
I20260927 22:47:21.771147 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:47:21.797117 3035 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:21.794762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.97.62:35751)
0927 22:47:21.794972 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:21.794979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:21.795366 (+ 387us) server_negotiation.cc:407] Connection header received
0927 22:47:21.795542 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:21.795546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:21.795625 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:21.795709 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:21.796874 (+ 1165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35192: BlockingRecv error: recv got EOF from 127.0.0.1:35192 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:47:21.799819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2905
W20260927 22:47:21.802213 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2905: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33811: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:47:21.802342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 2905
I20260927 22:47:21.814661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3039
W20260927 22:47:21.816942 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3039: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40331: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:47:21.817097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3039
I20260927 22:47:21.828486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3172
W20260927 22:47:21.831027 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3172: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:33145: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:47:21.831136 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3172
I20260927 22:47:21.842847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 2812
I20260927 22:47:21.843945 2874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:21.949954 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 2812
2026-09-27T22:47:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3204 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:22 dist-test-slave-kr25 krb5kdc[3317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:22 dist-test-slave-kr25 krb5kdc[3317](info): set up 2 sockets
Sep 27 22:47:22 dist-test-slave-kr25 krb5kdc[3317](info): commencing operation
krb5kdc: starting...
W20260927 22:47:22.939411 388 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:23 dist-test-slave-kr25 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:23.107725 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33037
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:23.407567 3330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:23.407835 3330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:23.407956 3330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:23.415125 3330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:23.415232 3330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:23.415301 3330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:23.415374 3330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:23.415454 3330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:23.426159 3330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:23.427464 3330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:23.428721 3330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:23.439707 3338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:23.439878 3335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:23.439710 3336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:23.440526 3330 server_base.cc:1034] running on GCE node
I20260927 22:47:23.441215 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:23.442519 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:23.443704 3330 hybrid_clock.cc:648] HybridClock initialized: now 1790549243443662 us; error 32 us; skew 500 ppm
I20260927 22:47:23.444018 3330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:23 dist-test-slave-kr25 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:23.446870 3330 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:23.448194 3330 webserver.cc:460] Webserver started at http://127.0.97.62:41095/ using document root <none> and password file <none>
I20260927 22:47:23.448808 3330 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:23.448902 3330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:23.449132 3330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:23.451936 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16"
format_stamp: "Formatted at 2026-09-27 22:47:23 on dist-test-slave-kr25"
server_key: "7a5c28766f6a9a4e7b3c49ccdd93cbe0"
server_key_iv: "4aa961373f50878e7c6ec79e9ccf4170"
server_key_version: "encryptionkey@0"
I20260927 22:47:23.452675 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16"
format_stamp: "Formatted at 2026-09-27 22:47:23 on dist-test-slave-kr25"
server_key: "7a5c28766f6a9a4e7b3c49ccdd93cbe0"
server_key_iv: "4aa961373f50878e7c6ec79e9ccf4170"
server_key_version: "encryptionkey@0"
I20260927 22:47:23.457425 3330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:23.460806 3345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:23.462057 3330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:47:23.462210 3330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16"
format_stamp: "Formatted at 2026-09-27 22:47:23 on dist-test-slave-kr25"
server_key: "7a5c28766f6a9a4e7b3c49ccdd93cbe0"
server_key_iv: "4aa961373f50878e7c6ec79e9ccf4170"
server_key_version: "encryptionkey@0"
I20260927 22:47:23.462386 3330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:23.484350 3330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:23.488117 3330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:23.488368 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:23.504757 3330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33037
I20260927 22:47:23.504787 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33037 every 8 connection(s)
I20260927 22:47:23.505980 3330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:23.509380 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:23.510859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3330
I20260927 22:47:23.511020 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:23.511639 388 external_mini_cluster.cc:1428] Setting key 5076025c4540b064511663e6f7b9e1ca
I20260927 22:47:23.520396 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16: Bootstrap starting.
I20260927 22:47:23.523926 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:23.525071 3397 log.cc:824] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:47:23 dist-test-slave-kr25 krb5kdc[3317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:23.527683 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16: No bootstrap required, opened a new log
I20260927 22:47:23.532794 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:23.513265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.0.97.62:33037)
0927 22:47:23.513973 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:23.513985 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:23.514029 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:47:23.515629 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:23.515651 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:23.516010 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:23.516301 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:23.518053 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:23.519314 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:23.520993 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:23.521640 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:23.521775 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:23.526060 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:23.526068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:23.526082 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:23.526120 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:23.528918 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:23.529568 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:23.529578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:23.529585 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:23.529755 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:23.530097 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:23.530101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:23.530103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:23.530619 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:23.530914 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:23.531340 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:47:23.531799 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":329,"threads_started":1}
I20260927 22:47:23.535120 3397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33037 } }
I20260927 22:47:23.535450 3397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:23.535567 3397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf504df3ebb47cdbca8ab6fedbf3e16, State: Initialized, Role: FOLLOWER
I20260927 22:47:23.536103 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33037 } }
I20260927 22:47:23.536339 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:47:23.536347 3397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:23.536454 3397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:23.536626 3397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:23.538651 3397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33037 } }
I20260927 22:47:23.539075 3397 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf504df3ebb47cdbca8ab6fedbf3e16; no voters:
I20260927 22:47:23.539638 3397 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:23.539727 3402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:23.541736 3397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:23.541901 3402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf504df3ebb47cdbca8ab6fedbf3e16, State: Running, Role: LEADER
I20260927 22:47:23.542451 3402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33037 } }
I20260927 22:47:23.545179 3403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33037 } } }
I20260927 22:47:23.545501 3403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:23.545961 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf504df3ebb47cdbca8ab6fedbf3e16. Latest consensus state: current_term: 1 leader_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf504df3ebb47cdbca8ab6fedbf3e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33037 } } }
I20260927 22:47:23.546241 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:23.559577 3411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:23.565393 3411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:23.590371 3411 catalog_manager.cc:1347] Generated new cluster ID: 9e617931ac864553804fa8fee21ccde9
I20260927 22:47:23.590503 3411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:23.614526 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:23.589747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.0.97.62:33037)
0927 22:47:23.589925 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:23.589932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:23.600882 (+ 10950us) server_negotiation.cc:407] Connection header received
0927 22:47:23.601046 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:23.601052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:23.601150 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:23.601287 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:23.602418 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:23.603550 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:23.605243 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:23.605888 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:23.606008 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:23.612751 (+ 6743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:23.612759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:23.612764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:23.613703 (+ 939us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:23.613765 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:23.613793 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:23.613915 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:23.614019 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 22:47:23.614164 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:47:23.632170 3411 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:23.633991 3411 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:23.641377 3411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf504df3ebb47cdbca8ab6fedbf3e16: Generated new TSK 0
I20260927 22:47:23.642390 3411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:47:23.654619 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3330
I20260927 22:47:23.655803 3392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:23.746090 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3330
2026-09-27T22:47:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1789 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:23 dist-test-slave-kr25 krb5kdc[3432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:23 dist-test-slave-kr25 krb5kdc[3432](info): set up 2 sockets
Sep 27 22:47:23 dist-test-slave-kr25 krb5kdc[3432](info): commencing operation
krb5kdc: starting...
W20260927 22:47:24.731501 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:24 dist-test-slave-kr25 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:24.904348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34679
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:25.205608 3445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:25.205906 3445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:25.206008 3445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:25.213166 3445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:25.213274 3445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:25.213327 3445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:25.213382 3445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:25.213423 3445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:25.222980 3445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:25.224277 3445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:25.225462 3445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:25.236071 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:25.236099 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:25.236132 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:25.236881 3445 server_base.cc:1034] running on GCE node
I20260927 22:47:25.237568 3445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:25.238867 3445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:25.240062 3445 hybrid_clock.cc:648] HybridClock initialized: now 1790549245240012 us; error 37 us; skew 500 ppm
I20260927 22:47:25.240381 3445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:25 dist-test-slave-kr25 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:25.243163 3445 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:25.244545 3445 webserver.cc:460] Webserver started at http://127.0.97.62:33857/ using document root <none> and password file <none>
I20260927 22:47:25.245158 3445 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:25.245267 3445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:25.245540 3445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:25.248371 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "e70b747a55b34ff68631d33674cecb77"
format_stamp: "Formatted at 2026-09-27 22:47:25 on dist-test-slave-kr25"
server_key: "c62060aa89d753b744bf1e0d98b7640d"
server_key_iv: "04e46e46739197fbd5ea67ad1772d93e"
server_key_version: "encryptionkey@0"
I20260927 22:47:25.249079 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "e70b747a55b34ff68631d33674cecb77"
format_stamp: "Formatted at 2026-09-27 22:47:25 on dist-test-slave-kr25"
server_key: "c62060aa89d753b744bf1e0d98b7640d"
server_key_iv: "04e46e46739197fbd5ea67ad1772d93e"
server_key_version: "encryptionkey@0"
I20260927 22:47:25.253793 3445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:25.256978 3460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:25.258206 3445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:47:25.258350 3445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "e70b747a55b34ff68631d33674cecb77"
format_stamp: "Formatted at 2026-09-27 22:47:25 on dist-test-slave-kr25"
server_key: "c62060aa89d753b744bf1e0d98b7640d"
server_key_iv: "04e46e46739197fbd5ea67ad1772d93e"
server_key_version: "encryptionkey@0"
I20260927 22:47:25.258549 3445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:25.290701 3445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:25.294960 3445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:25.295230 3445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:25.311342 3445 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34679
I20260927 22:47:25.311365 3511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34679 every 8 connection(s)
I20260927 22:47:25.312709 3445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:25.316232 3512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:25.319674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3445
I20260927 22:47:25.319839 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:25.320426 388 external_mini_cluster.cc:1428] Setting key ec0a4a80a3fd799d6e953427b29d4e27
I20260927 22:47:25.328778 3512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77: Bootstrap starting.
I20260927 22:47:25.332069 3512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:25.332978 3512 log.cc:824] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:47:25 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:25.335798 3512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77: No bootstrap required, opened a new log
I20260927 22:47:25.341421 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:25.322206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.97.62:34679)
0927 22:47:25.322942 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:25.322958 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:25.323007 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:47:25.324603 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:25.324631 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:25.325153 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:25.325590 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:25.326737 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:25.328620 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:25.329863 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:25.330885 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:25.331048 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:25.334787 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:25.334795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:25.334809 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:25.334844 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:25.337701 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:25.338341 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:25.338347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:25.338353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:25.338511 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:25.338859 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:25.338863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:25.338866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:25.339366 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:25.339707 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:25.340070 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 22:47:25.340450 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":296,"threads_started":1}
I20260927 22:47:25.343420 3512 raft_consensus.cc:348] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70b747a55b34ff68631d33674cecb77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34679 } }
I20260927 22:47:25.343845 3512 raft_consensus.cc:374] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:25.343942 3512 raft_consensus.cc:729] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70b747a55b34ff68631d33674cecb77, State: Initialized, Role: FOLLOWER
I20260927 22:47:25.344446 3512 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70b747a55b34ff68631d33674cecb77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34679 } }
I20260927 22:47:25.344642 3512 raft_consensus.cc:388] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:25.344764 3512 raft_consensus.cc:482] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:25.344895 3512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:25.346982 3512 raft_consensus.cc:504] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70b747a55b34ff68631d33674cecb77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34679 } }
I20260927 22:47:25.347404 3512 leader_election.cc:302] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e70b747a55b34ff68631d33674cecb77; no voters:
I20260927 22:47:25.347947 3512 leader_election.cc:288] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:25.348059 3517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:25.348644 3517 raft_consensus.cc:686] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [term 1 LEADER]: Becoming Leader. State: Replica: e70b747a55b34ff68631d33674cecb77, State: Running, Role: LEADER
I20260927 22:47:25.349098 3517 consensus_queue.cc:237] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70b747a55b34ff68631d33674cecb77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34679 } }
I20260927 22:47:25.349697 3512 sys_catalog.cc:564] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:25.351608 3519 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70b747a55b34ff68631d33674cecb77. Latest consensus state: current_term: 1 leader_uuid: "e70b747a55b34ff68631d33674cecb77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70b747a55b34ff68631d33674cecb77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34679 } } }
I20260927 22:47:25.351928 3519 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:25.352840 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e70b747a55b34ff68631d33674cecb77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70b747a55b34ff68631d33674cecb77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34679 } } }
I20260927 22:47:25.353159 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:25.354781 3524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:25.359771 3524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:25.369117 3524 catalog_manager.cc:1347] Generated new cluster ID: d4dc037c66db4ad0a8fe860da80d0996
I20260927 22:47:25.369252 3524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:25.386323 3524 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:25.388221 3524 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:25.405086 3524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e70b747a55b34ff68631d33674cecb77: Generated new TSK 0
I20260927 22:47:25.406139 3524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:25.452297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34679
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:25.756083 3538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:25.756361 3538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:25.756511 3538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:25.763700 3538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:25.763813 3538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:25.763967 3538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:25.773815 3538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:25.775101 3538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:25.776463 3538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:25.788899 3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:25.788936 3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:25.788848 3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:25.789649 3538 server_base.cc:1034] running on GCE node
I20260927 22:47:25.790217 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:25.791093 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:25.792303 3538 hybrid_clock.cc:648] HybridClock initialized: now 1790549245792255 us; error 35 us; skew 500 ppm
I20260927 22:47:25.792616 3538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:25 dist-test-slave-kr25 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:25.795714 3538 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:25.797087 3538 webserver.cc:460] Webserver started at http://127.0.97.1:33425/ using document root <none> and password file <none>
I20260927 22:47:25.797727 3538 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:25.797843 3538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:25.798130 3538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:25.801071 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "814245db73a84e178e08075fcd7ceced"
format_stamp: "Formatted at 2026-09-27 22:47:25 on dist-test-slave-kr25"
server_key: "d482de56ebb7f628512541b65191df9e"
server_key_iv: "c1e6766411a07082b805f46ea8aedc5e"
server_key_version: "encryptionkey@0"
I20260927 22:47:25.801832 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "814245db73a84e178e08075fcd7ceced"
format_stamp: "Formatted at 2026-09-27 22:47:25 on dist-test-slave-kr25"
server_key: "d482de56ebb7f628512541b65191df9e"
server_key_iv: "c1e6766411a07082b805f46ea8aedc5e"
server_key_version: "encryptionkey@0"
I20260927 22:47:25.806769 3538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:47:25.809964 3553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:25.811225 3538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:25.811363 3538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "814245db73a84e178e08075fcd7ceced"
format_stamp: "Formatted at 2026-09-27 22:47:25 on dist-test-slave-kr25"
server_key: "d482de56ebb7f628512541b65191df9e"
server_key_iv: "c1e6766411a07082b805f46ea8aedc5e"
server_key_version: "encryptionkey@0"
I20260927 22:47:25.811565 3538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:25.822050 3538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:25.825842 3538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:25.826102 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:25.826902 3538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:25.828125 3538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:25.828195 3538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:25.828308 3538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:25.828358 3538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:25.855902 3538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36693
I20260927 22:47:25.855924 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36693 every 8 connection(s)
I20260927 22:47:25.857151 3538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:25.859953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3538
I20260927 22:47:25.860131 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:25.860939 388 external_mini_cluster.cc:1428] Setting key fea8f47cc19ddc027b0f6b9c7bbbf5b4
Sep 27 22:47:25 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:25.877377 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:25.859419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49033 (local address 127.0.97.62:34679)
0927 22:47:25.859905 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:25.859911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:25.860920 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 22:47:25.862889 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:25.862911 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:25.862990 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:25.863079 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:25.864948 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:25.865621 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:25.867672 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:25.868058 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:25.868190 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:25.871502 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:25.871508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:25.871513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:25.871548 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:25.874382 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:25.875030 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:25.875036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:25.875040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:25.875176 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:25.875561 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:25.875566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:25.875570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:25.875923 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:25.876075 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:25.876810 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 22:47:25.877020 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":228,"threads_started":1}
I20260927 22:47:25.878315 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:25.859985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34679 (local address 127.0.97.1:49033)
0927 22:47:25.860748 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:25.860807 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:25.862617 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:25.863337 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:25.863349 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:25.864026 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:25.864798 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:25.864820 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:25.865759 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:25.865764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:25.867555 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:25.867568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:25.868246 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:25.868250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:25.868435 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:25.869048 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:25.869069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:25.871252 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:25.874501 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:25.874507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:25.874520 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:25.874908 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:25.875268 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:25.875271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:25.875273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:25.875458 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:25.876102 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:25.876109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:25.876431 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 22:47:25.876683 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:47:25.877128 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:25.879932 3666 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34679
I20260927 22:47:25.880235 3666 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:25.880970 3666 heartbeater.cc:511] Master 127.0.97.62:34679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:47:25.883416 3477 ts_manager.cc:194] Registered new tserver with Master: 814245db73a84e178e08075fcd7ceced (127.0.97.1:36693)
I20260927 22:47:25.885330 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:25.902966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34679
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:26.206267 3672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:26.206527 3672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:26.206616 3672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:26.213662 3672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:26.213764 3672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:26.213878 3672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:26.223701 3672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:26.224885 3672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:26.226105 3672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:26.237941 3677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:26.238023 3678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:26.237973 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:26.239024 3672 server_base.cc:1034] running on GCE node
I20260927 22:47:26.239543 3672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:26.240332 3672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:26.241528 3672 hybrid_clock.cc:648] HybridClock initialized: now 1790549246241464 us; error 49 us; skew 500 ppm
I20260927 22:47:26.241802 3672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:26.244664 3672 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:26.245910 3672 webserver.cc:460] Webserver started at http://127.0.97.2:41731/ using document root <none> and password file <none>
I20260927 22:47:26.246563 3672 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:26.246681 3672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:26.246950 3672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:26.250090 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "a91ec1c303704f8e8d661ca8ae8f1f02"
format_stamp: "Formatted at 2026-09-27 22:47:26 on dist-test-slave-kr25"
server_key: "c9cdf91e135cf35eb65ee938d79bebc1"
server_key_iv: "63a57f9f00e486a19d641e18f3476ed4"
server_key_version: "encryptionkey@0"
I20260927 22:47:26.251077 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a91ec1c303704f8e8d661ca8ae8f1f02"
format_stamp: "Formatted at 2026-09-27 22:47:26 on dist-test-slave-kr25"
server_key: "c9cdf91e135cf35eb65ee938d79bebc1"
server_key_iv: "63a57f9f00e486a19d641e18f3476ed4"
server_key_version: "encryptionkey@0"
I20260927 22:47:26.257495 3672 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 22:47:26.260648 3687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:26.261991 3672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:47:26.262112 3672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "a91ec1c303704f8e8d661ca8ae8f1f02"
format_stamp: "Formatted at 2026-09-27 22:47:26 on dist-test-slave-kr25"
server_key: "c9cdf91e135cf35eb65ee938d79bebc1"
server_key_iv: "63a57f9f00e486a19d641e18f3476ed4"
server_key_version: "encryptionkey@0"
I20260927 22:47:26.262275 3672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:26.278863 3672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:26.282403 3672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:26.282634 3672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:26.283520 3672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:26.284843 3672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:26.284914 3672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:26.284994 3672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:26.285034 3672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:26.313778 3672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35555
I20260927 22:47:26.313825 3799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35555 every 8 connection(s)
I20260927 22:47:26.315053 3672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:26.317971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3672
I20260927 22:47:26.318135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:26.318687 388 external_mini_cluster.cc:1428] Setting key e3e7d3343976d9749c74c312fdb1c1eb
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:47:26.336537 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.317659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56583 (local address 127.0.97.62:34679)
0927 22:47:26.318520 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:26.318527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:26.319052 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:47:26.321964 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:26.321970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:26.322045 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:26.322142 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:26.324005 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.325104 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.326801 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.327259 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.327365 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.330986 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:26.330991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:26.330995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:26.331027 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:26.333538 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.334376 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.334380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.334382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.334489 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.334913 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.334916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.334918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.335156 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:26.335259 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:26.336115 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 22:47:26.336317 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720}
I20260927 22:47:26.337568 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.318099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34679 (local address 127.0.97.2:56583)
0927 22:47:26.318910 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:26.318953 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:26.321667 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:26.322398 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:26.322408 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:26.323064 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:26.323833 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:26.323857 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.325259 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.325264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:26.326661 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:26.326674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.327495 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.327502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:26.327746 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.328366 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:26.328391 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:26.330773 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:26.333676 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:26.333685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:26.333703 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:26.334272 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:26.334582 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:26.334587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:26.334591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:26.334816 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:26.335283 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:26.335292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:26.335748 (+ 456us) client_negotiation.cc:769] Sending connection context
0927 22:47:26.336049 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 22:47:26.336516 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":329,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:47:26.338932 3800 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34679
I20260927 22:47:26.339198 3800 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:26.339936 3800 heartbeater.cc:511] Master 127.0.97.62:34679 requested a full tablet report, sending...
I20260927 22:47:26.341270 3477 ts_manager.cc:194] Registered new tserver with Master: a91ec1c303704f8e8d661ca8ae8f1f02 (127.0.97.2:35555)
I20260927 22:47:26.342058 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:26.359232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34679
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:26.658746 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:26.659015 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:26.659132 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:26.668182 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:26.668314 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:26.668494 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:26.678663 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:26.679885 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:26.681075 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:26.693286 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:26.693372 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:26.693626 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:26.694219 3805 server_base.cc:1034] running on GCE node
I20260927 22:47:26.694779 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:26.695669 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:26.696861 3805 hybrid_clock.cc:648] HybridClock initialized: now 1790549246696812 us; error 42 us; skew 500 ppm
I20260927 22:47:26.697168 3805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:26.700037 3805 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:26.701304 3805 webserver.cc:460] Webserver started at http://127.0.97.3:38439/ using document root <none> and password file <none>
I20260927 22:47:26.701922 3805 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:26.702039 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:26.702294 3805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:26.704995 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "e6f85272e8524425a765ceb7d8640bc6"
format_stamp: "Formatted at 2026-09-27 22:47:26 on dist-test-slave-kr25"
server_key: "7ab790a81cb3f28d0804e91922fcc2f9"
server_key_iv: "cd867033adc30d435ff0fcf72ceacc65"
server_key_version: "encryptionkey@0"
I20260927 22:47:26.705714 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e6f85272e8524425a765ceb7d8640bc6"
format_stamp: "Formatted at 2026-09-27 22:47:26 on dist-test-slave-kr25"
server_key: "7ab790a81cb3f28d0804e91922fcc2f9"
server_key_iv: "cd867033adc30d435ff0fcf72ceacc65"
server_key_version: "encryptionkey@0"
I20260927 22:47:26.710201 3805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:47:26.713228 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:26.714391 3805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 22:47:26.714527 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "e6f85272e8524425a765ceb7d8640bc6"
format_stamp: "Formatted at 2026-09-27 22:47:26 on dist-test-slave-kr25"
server_key: "7ab790a81cb3f28d0804e91922fcc2f9"
server_key_iv: "cd867033adc30d435ff0fcf72ceacc65"
server_key_version: "encryptionkey@0"
I20260927 22:47:26.714695 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:26.725132 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:26.728797 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:26.729041 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:26.729831 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:26.730979 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:26.731086 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:26.731190 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:26.731240 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:26.759809 3805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46611
I20260927 22:47:26.759872 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46611 every 8 connection(s)
I20260927 22:47:26.761027 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:26.763285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3805
I20260927 22:47:26.763471 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:26.763988 388 external_mini_cluster.cc:1428] Setting key 509dba823699d8a7222ec33308d6e8d3
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:26.780308 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.763613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35637 (local address 127.0.97.62:34679)
0927 22:47:26.763803 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:26.763808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:26.765100 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 22:47:26.767350 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:26.767356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:26.767471 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:26.767564 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:26.769029 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.769783 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.771343 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.771850 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.771982 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.775275 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:26.775280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:26.775284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:26.775316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:26.777607 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.778299 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.778302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.778304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.778448 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.778891 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.778894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.778896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.779181 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:26.779278 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:26.779933 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 22:47:26.780080 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:26.781167 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.764182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34679 (local address 127.0.97.3:35637)
0927 22:47:26.764931 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:26.764978 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:26.767092 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:26.767803 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:26.767811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:26.768285 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:26.768867 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:26.768885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.769937 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.769941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:26.771169 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:26.771186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.772065 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.772073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:26.772313 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.772893 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:26.772920 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:26.775058 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:26.777722 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:26.777737 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:26.777753 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:26.778198 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:26.778567 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:26.778572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:26.778575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:26.778793 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:26.779285 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:26.779291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:26.779642 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 22:47:26.779859 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 22:47:26.780210 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":285,"threads_started":1}
I20260927 22:47:26.782259 3933 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34679
I20260927 22:47:26.782526 3933 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:26.783138 3933 heartbeater.cc:511] Master 127.0.97.62:34679 requested a full tablet report, sending...
I20260927 22:47:26.784391 3477 ts_manager.cc:194] Registered new tserver with Master: e6f85272e8524425a765ceb7d8640bc6 (127.0.97.3:46611)
I20260927 22:47:26.785261 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35637
I20260927 22:47:26.790858 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:26.832235 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.819826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.0.97.62:34679)
0927 22:47:26.820016 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:26.820022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:26.820368 (+ 346us) server_negotiation.cc:407] Connection header received
0927 22:47:26.820544 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:26.820549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:26.820622 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:26.820703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:26.821979 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.822673 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.823893 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.824298 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.824406 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.827825 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:26.827831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:26.827834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:26.827875 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:26.830144 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.830689 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.830693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.830695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.830819 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.831143 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.831147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.831149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.831489 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:26.831670 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:26.831852 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 22:47:26.832019 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:47:26.832736 3477 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45076
I20260927 22:47:26.834728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3538
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:26.848402 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.835584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.0.97.1:36693)
0927 22:47:26.836019 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:26.836025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:26.836048 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:47:26.836099 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:26.836104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:26.836298 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:26.836411 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:26.837424 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.838455 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.839657 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.840165 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.840250 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.843537 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:26.843547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:26.843560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:26.843595 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:26.846003 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.846591 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.846597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.846599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.846775 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.847115 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.847118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.847120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.847427 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:26.847672 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:26.847873 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 22:47:26.848070 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":205,"threads_started":1}
W20260927 22:47:26.849562 3661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46836
W20260927 22:47:26.850517 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:26.850693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3538
I20260927 22:47:26.862639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3672
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:26.876932 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.863453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.0.97.2:35555)
0927 22:47:26.863935 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:26.863943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:26.863969 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:47:26.864042 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:26.864049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:26.864290 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:26.864424 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:26.865533 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.866555 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.867758 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.868280 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.868370 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.871867 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:26.871878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:26.871891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:26.871928 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:26.874532 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.875047 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.875053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.875055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.875176 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.875589 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.875593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.875596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.875908 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:26.876125 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:26.876383 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:47:26.876638 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
W20260927 22:47:26.878176 3795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33202
W20260927 22:47:26.879398 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:26.879601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3672
I20260927 22:47:26.891425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3805
Sep 27 22:47:26 dist-test-slave-kr25 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:26.906657 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:26.892526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.0.97.3:46611)
0927 22:47:26.893132 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:26.893138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:26.893177 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:47:26.893234 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:26.893240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:26.893444 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:26.893613 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:26.894835 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.895957 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.897122 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:26.897685 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:26.897786 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:26.901681 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:26.901692 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:26.901706 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:26.901740 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:26.904376 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.904940 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.904946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.904949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.905056 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:26.905335 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:26.905339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:26.905341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:26.905717 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:26.905941 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:26.906116 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 22:47:26.906308 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":327,"threads_started":1}
W20260927 22:47:26.907907 3928 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51868
W20260927 22:47:26.908955 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:47:26.909123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3805
I20260927 22:47:26.920665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3445
I20260927 22:47:26.921911 3507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:27.020903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3445
2026-09-27T22:47:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3279 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:27 dist-test-slave-kr25 krb5kdc[3950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:27 dist-test-slave-kr25 krb5kdc[3950](info): set up 2 sockets
Sep 27 22:47:27 dist-test-slave-kr25 krb5kdc[3950](info): commencing operation
krb5kdc: starting...
W20260927 22:47:28.011518 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:28 dist-test-slave-kr25 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:28.180567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42945
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:28.481050 3963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:28.481350 3963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:28.481484 3963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:28.488538 3963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:28.488649 3963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:28.488730 3963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:28.488811 3963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:28.488873 3963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:28.498330 3963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:28.499563 3963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:28.500787 3963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:28.511335 3969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:28.511425 3968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:28.511363 3971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:28.512006 3963 server_base.cc:1034] running on GCE node
I20260927 22:47:28.512634 3963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:28.513899 3963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:28.515084 3963 hybrid_clock.cc:648] HybridClock initialized: now 1790549248515040 us; error 34 us; skew 500 ppm
I20260927 22:47:28.515391 3963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:28 dist-test-slave-kr25 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:28.518225 3963 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:28.519686 3963 webserver.cc:460] Webserver started at http://127.0.97.62:34823/ using document root <none> and password file <none>
I20260927 22:47:28.520311 3963 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:28.520407 3963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:28.520704 3963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:28.523550 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "3d105c75a6bf47589c6faed413e70982"
format_stamp: "Formatted at 2026-09-27 22:47:28 on dist-test-slave-kr25"
server_key: "06a5e07386a7b8566d394a991fe6ab3d"
server_key_iv: "6b3f65aec80e772f322c90ae5fb02d9f"
server_key_version: "encryptionkey@0"
I20260927 22:47:28.524297 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "3d105c75a6bf47589c6faed413e70982"
format_stamp: "Formatted at 2026-09-27 22:47:28 on dist-test-slave-kr25"
server_key: "06a5e07386a7b8566d394a991fe6ab3d"
server_key_iv: "6b3f65aec80e772f322c90ae5fb02d9f"
server_key_version: "encryptionkey@0"
I20260927 22:47:28.529384 3963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:47:28.532639 3978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:28.533948 3963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:47:28.534083 3963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "3d105c75a6bf47589c6faed413e70982"
format_stamp: "Formatted at 2026-09-27 22:47:28 on dist-test-slave-kr25"
server_key: "06a5e07386a7b8566d394a991fe6ab3d"
server_key_iv: "6b3f65aec80e772f322c90ae5fb02d9f"
server_key_version: "encryptionkey@0"
I20260927 22:47:28.534255 3963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:28.549058 3963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:28.552902 3963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:28.553155 3963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:28.569633 3963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42945
I20260927 22:47:28.569654 4029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42945 every 8 connection(s)
I20260927 22:47:28.570906 3963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:28.574102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 3963
I20260927 22:47:28.574318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:28.574743 4030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:28.575110 388 external_mini_cluster.cc:1428] Setting key 2c8fca59ac8d927c471360b335cc8117
I20260927 22:47:28.588160 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982: Bootstrap starting.
Sep 27 22:47:28 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:28.591105 4030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:28.592103 4030 log.cc:824] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:28.594667 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982: No bootstrap required, opened a new log
I20260927 22:47:28.597158 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:28.576977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.0.97.62:42945)
0927 22:47:28.577725 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:28.577736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:28.577776 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:47:28.579240 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:28.579261 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:28.579664 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:28.579966 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:28.581642 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:28.582839 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:28.584502 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:28.585180 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:28.585316 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:28.589813 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:28.589826 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:28.589845 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:28.589887 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:28.593063 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:28.593775 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:28.593783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:28.593789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:28.593951 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:28.594325 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:28.594330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:28.594333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:28.594894 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:28.595204 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:28.595680 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 22:47:28.596093 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260927 22:47:28.602095 4030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d105c75a6bf47589c6faed413e70982" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42945 } }
I20260927 22:47:28.602502 4030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:28.602617 4030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d105c75a6bf47589c6faed413e70982, State: Initialized, Role: FOLLOWER
I20260927 22:47:28.603193 4030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d105c75a6bf47589c6faed413e70982" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42945 } }
I20260927 22:47:28.603451 4030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:28.603580 4030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:28.603720 4030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:28.605746 4030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d105c75a6bf47589c6faed413e70982" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42945 } }
I20260927 22:47:28.606172 4030 leader_election.cc:302] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d105c75a6bf47589c6faed413e70982; no voters:
I20260927 22:47:28.606742 4030 leader_election.cc:288] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:28.606848 4035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:28.607779 4035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [term 1 LEADER]: Becoming Leader. State: Replica: 3d105c75a6bf47589c6faed413e70982, State: Running, Role: LEADER
I20260927 22:47:28.608245 4035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d105c75a6bf47589c6faed413e70982" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42945 } }
I20260927 22:47:28.608690 4030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:28.610813 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d105c75a6bf47589c6faed413e70982. Latest consensus state: current_term: 1 leader_uuid: "3d105c75a6bf47589c6faed413e70982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d105c75a6bf47589c6faed413e70982" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42945 } } }
I20260927 22:47:28.611164 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:28.611917 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d105c75a6bf47589c6faed413e70982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d105c75a6bf47589c6faed413e70982" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42945 } } }
I20260927 22:47:28.612263 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:28.617080 4041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:28.622197 4041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:28.629971 4041 catalog_manager.cc:1347] Generated new cluster ID: 4159be4a061943e280df6a9ec9d3ef75
I20260927 22:47:28.630067 4041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:28.645319 4041 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:28.646857 4041 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:28.653409 4041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d105c75a6bf47589c6faed413e70982: Generated new TSK 0
I20260927 22:47:28.654186 4041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:28.710791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42945
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:29.043838 4056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:29.044149 4056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:29.044293 4056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:29.051507 4056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:29.051621 4056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:29.051777 4056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:29.061530 4056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:29.062747 4056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:29.064020 4056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:29.076395 4062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:29.076428 4061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:29.076395 4064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:29.077034 4056 server_base.cc:1034] running on GCE node
I20260927 22:47:29.077526 4056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:29.078341 4056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:29.079509 4056 hybrid_clock.cc:648] HybridClock initialized: now 1790549249079452 us; error 48 us; skew 500 ppm
I20260927 22:47:29.079777 4056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:29 dist-test-slave-kr25 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:29.088603 4056 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:29.089972 4056 webserver.cc:460] Webserver started at http://127.0.97.1:36695/ using document root <none> and password file <none>
I20260927 22:47:29.090631 4056 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:29.090741 4056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:29.091018 4056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:29.093870 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "b4f8a8162bb8477a9666f0d87736dd7c"
format_stamp: "Formatted at 2026-09-27 22:47:29 on dist-test-slave-kr25"
server_key: "5d093c870b0fed809ffb90540a109fca"
server_key_iv: "9642eaccf0b863b39b0ac53e0153a111"
server_key_version: "encryptionkey@0"
I20260927 22:47:29.094579 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b4f8a8162bb8477a9666f0d87736dd7c"
format_stamp: "Formatted at 2026-09-27 22:47:29 on dist-test-slave-kr25"
server_key: "5d093c870b0fed809ffb90540a109fca"
server_key_iv: "9642eaccf0b863b39b0ac53e0153a111"
server_key_version: "encryptionkey@0"
I20260927 22:47:29.099382 4056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 22:47:29.102530 4071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:29.103785 4056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:29.103915 4056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "b4f8a8162bb8477a9666f0d87736dd7c"
format_stamp: "Formatted at 2026-09-27 22:47:29 on dist-test-slave-kr25"
server_key: "5d093c870b0fed809ffb90540a109fca"
server_key_iv: "9642eaccf0b863b39b0ac53e0153a111"
server_key_version: "encryptionkey@0"
I20260927 22:47:29.104091 4056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:29.116345 4056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:29.119936 4056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:29.120177 4056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:29.120968 4056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:29.122124 4056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:29.122200 4056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:29.122304 4056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:29.122354 4056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:29.150602 4056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38537
I20260927 22:47:29.150648 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38537 every 8 connection(s)
I20260927 22:47:29.151837 4056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:29.158823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 4056
I20260927 22:47:29.159034 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:29.159778 388 external_mini_cluster.cc:1428] Setting key 772316ad2125c7aab5d1ba7e203ab5e0
Sep 27 22:47:29 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:29.171988 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:29.154365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54591 (local address 127.0.97.62:42945)
0927 22:47:29.154743 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:29.154747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:29.155786 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 22:47:29.157563 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:29.157568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:29.157642 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:29.157756 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:29.159678 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.160483 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.162204 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.162580 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.162683 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:29.166198 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:29.166204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:29.166209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:29.166247 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:29.169161 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:29.169817 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:29.169821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:29.169824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:29.169950 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:29.170314 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:29.170318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:29.170322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:29.170662 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:29.170791 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:29.171425 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 22:47:29.171665 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":192,"threads_started":1}
I20260927 22:47:29.172809 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:29.154925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42945 (local address 127.0.97.1:54591)
0927 22:47:29.155646 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:29.155685 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:29.157239 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:29.157997 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:29.158008 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:29.158702 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:29.159505 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:29.159531 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.160632 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.160638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:29.162079 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:29.162091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.162779 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.162785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:29.163007 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:29.163625 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:29.163646 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:29.165974 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:29.169289 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:29.169295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:29.169308 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:29.169707 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:29.170033 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:29.170036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:29.170039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:29.170220 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:29.170764 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:29.170769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:29.171083 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 22:47:29.171309 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 22:47:29.171728 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":284,"threads_started":1}
I20260927 22:47:29.174361 4184 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42945
I20260927 22:47:29.174641 4184 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:29.175307 4184 heartbeater.cc:511] Master 127.0.97.62:42945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:29.177345 3995 ts_manager.cc:194] Registered new tserver with Master: b4f8a8162bb8477a9666f0d87736dd7c (127.0.97.1:38537)
I20260927 22:47:29.178882 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:29.199980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42945
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:29.500602 4190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:29.500876 4190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:29.501015 4190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:29.508016 4190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:29.508127 4190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:29.508257 4190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:29.517602 4190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:29.518823 4190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:29.520047 4190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:29.532272 4195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:29.532192 4196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:29.532222 4198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:29.533417 4190 server_base.cc:1034] running on GCE node
I20260927 22:47:29.533953 4190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:29.534797 4190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:29.535979 4190 hybrid_clock.cc:648] HybridClock initialized: now 1790549249535928 us; error 41 us; skew 500 ppm
I20260927 22:47:29.536279 4190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:29 dist-test-slave-kr25 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:29.539259 4190 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:29.540625 4190 webserver.cc:460] Webserver started at http://127.0.97.2:34537/ using document root <none> and password file <none>
I20260927 22:47:29.541232 4190 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:29.541326 4190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:29.541570 4190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:29.544296 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "c82a2b94465746218a9087e48053354c"
format_stamp: "Formatted at 2026-09-27 22:47:29 on dist-test-slave-kr25"
server_key: "1459ff6a49f2c2d060c54dff0b35113d"
server_key_iv: "bd9bc3f11692a7b911e6ce4a83fc0ae6"
server_key_version: "encryptionkey@0"
I20260927 22:47:29.544973 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c82a2b94465746218a9087e48053354c"
format_stamp: "Formatted at 2026-09-27 22:47:29 on dist-test-slave-kr25"
server_key: "1459ff6a49f2c2d060c54dff0b35113d"
server_key_iv: "bd9bc3f11692a7b911e6ce4a83fc0ae6"
server_key_version: "encryptionkey@0"
I20260927 22:47:29.549780 4190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:47:29.552950 4205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:29.554178 4190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:47:29.554309 4190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "c82a2b94465746218a9087e48053354c"
format_stamp: "Formatted at 2026-09-27 22:47:29 on dist-test-slave-kr25"
server_key: "1459ff6a49f2c2d060c54dff0b35113d"
server_key_iv: "bd9bc3f11692a7b911e6ce4a83fc0ae6"
server_key_version: "encryptionkey@0"
I20260927 22:47:29.554474 4190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:29.571486 4190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:29.575398 4190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:29.575683 4190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:29.576663 4190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:29.577960 4190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:29.578032 4190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:29.578148 4190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:29.578197 4190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:29.606913 4190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41445
I20260927 22:47:29.606978 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41445 every 8 connection(s)
I20260927 22:47:29.608256 4190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:29.613914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 4190
I20260927 22:47:29.614095 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:29.614624 388 external_mini_cluster.cc:1428] Setting key 3e73d54063d8e8fa4aef67d5211f3b17
Sep 27 22:47:29 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:29.629314 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:29.610707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55685 (local address 127.0.97.62:42945)
0927 22:47:29.610873 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:29.610878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:29.612026 (+ 1148us) server_negotiation.cc:407] Connection header received
0927 22:47:29.613815 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:29.613820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:29.613899 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:29.613987 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:29.615808 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.616712 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.618886 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.619300 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.619400 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:29.623395 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:29.623399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:29.623402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:29.623453 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:29.626376 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:29.627094 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:29.627098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:29.627101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:29.627247 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:29.627689 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:29.627693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:29.627696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:29.627980 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:29.628109 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:29.628782 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:47:29.628978 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:47:29.630259 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:29.611176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42945 (local address 127.0.97.2:55685)
0927 22:47:29.611872 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:29.611915 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:29.613504 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:29.614264 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:29.614276 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:29.614976 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:29.615631 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:29.615650 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.616866 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.616871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:29.618726 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:29.618740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:29.619565 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:29.619572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:29.619842 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:29.620508 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:29.620535 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:29.623180 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:29.626540 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:29.626547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:29.626559 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:29.626975 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:29.627331 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:29.627334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:29.627337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:29.627584 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:29.628120 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:29.628126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:29.628427 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 22:47:29.628675 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:47:29.629039 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":286,"threads_started":1}
I20260927 22:47:29.631708 4318 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42945
I20260927 22:47:29.631996 4318 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:29.632712 4318 heartbeater.cc:511] Master 127.0.97.62:42945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:47:29.634120 3995 ts_manager.cc:194] Registered new tserver with Master: c82a2b94465746218a9087e48053354c (127.0.97.2:41445)
I20260927 22:47:29.634929 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:29.656010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42945
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:29.963622 4323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:29.963904 4323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:29.964036 4323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:29.971261 4323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:29.971382 4323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:29.971554 4323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:29.981268 4323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:29.982496 4323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:29.983817 4323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:29.996048 4328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:29.996006 4331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:29.996098 4329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:29.997196 4323 server_base.cc:1034] running on GCE node
I20260927 22:47:29.997745 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:29.998673 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:29.999863 4323 hybrid_clock.cc:648] HybridClock initialized: now 1790549249999816 us; error 37 us; skew 500 ppm
I20260927 22:47:30.000151 4323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:29 dist-test-slave-kr25 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:30.003113 4323 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:30.004696 4323 webserver.cc:460] Webserver started at http://127.0.97.3:35921/ using document root <none> and password file <none>
I20260927 22:47:30.005350 4323 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:30.005463 4323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:30.005745 4323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:30.008747 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "8a31b37de2914887a12a83f302cfc090"
format_stamp: "Formatted at 2026-09-27 22:47:30 on dist-test-slave-kr25"
server_key: "1a3ea93d20174a3e0de72f36feb0e4ec"
server_key_iv: "e4e83d30cc3044b721da28695027cb0e"
server_key_version: "encryptionkey@0"
I20260927 22:47:30.009517 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8a31b37de2914887a12a83f302cfc090"
format_stamp: "Formatted at 2026-09-27 22:47:30 on dist-test-slave-kr25"
server_key: "1a3ea93d20174a3e0de72f36feb0e4ec"
server_key_iv: "e4e83d30cc3044b721da28695027cb0e"
server_key_version: "encryptionkey@0"
I20260927 22:47:30.014569 4323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:47:30.017875 4338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:30.019258 4323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:30.019378 4323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "8a31b37de2914887a12a83f302cfc090"
format_stamp: "Formatted at 2026-09-27 22:47:30 on dist-test-slave-kr25"
server_key: "1a3ea93d20174a3e0de72f36feb0e4ec"
server_key_iv: "e4e83d30cc3044b721da28695027cb0e"
server_key_version: "encryptionkey@0"
I20260927 22:47:30.019589 4323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:30.047101 4323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:30.051049 4323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:30.051294 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:30.052315 4323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:30.053645 4323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:30.053715 4323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:30.053820 4323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:30.053874 4323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:30.081755 4323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41425
I20260927 22:47:30.081805 4450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41425 every 8 connection(s)
I20260927 22:47:30.082931 4323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:30.092460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 4323
I20260927 22:47:30.092656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:30.093163 388 external_mini_cluster.cc:1428] Setting key 301483170a3d601427cd051cd49acec6
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:30.101269 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:30.085351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56815 (local address 127.0.97.62:42945)
0927 22:47:30.085555 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:30.085561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:30.086712 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 22:47:30.088440 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:30.088444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:30.088508 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:30.088601 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:30.089964 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.090679 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.092185 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.092776 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.092900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:30.096228 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:30.096234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:30.096237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:30.096268 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:30.098488 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.099141 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.099147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.099149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.099259 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.099702 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.099707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.099710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.099973 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:30.100099 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:30.100746 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 22:47:30.100944 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:47:30.102156 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:30.085848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42945 (local address 127.0.97.3:56815)
0927 22:47:30.086557 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:30.086595 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:30.088206 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:30.088796 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:30.088804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:30.089295 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:30.089837 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:30.089853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.090793 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.090796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:30.092050 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:30.092062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.092962 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.092967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:30.093162 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:30.093811 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:30.093832 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:30.096031 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:30.098625 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:30.098634 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:30.098649 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:30.099018 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:30.099352 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:30.099357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:30.099361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:30.099595 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:30.100096 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:30.100104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:30.100413 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 22:47:30.100631 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 22:47:30.101125 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":300,"threads_started":1}
I20260927 22:47:30.103392 4451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42945
I20260927 22:47:30.103689 4451 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:30.104333 4451 heartbeater.cc:511] Master 127.0.97.62:42945 requested a full tablet report, sending...
I20260927 22:47:30.105592 3995 ts_manager.cc:194] Registered new tserver with Master: 8a31b37de2914887a12a83f302cfc090 (127.0.97.3:41425)
I20260927 22:47:30.106439 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56815
I20260927 22:47:30.108001 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:30.151502 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:30.137278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.0.97.1:38537)
0927 22:47:30.137805 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:30.137814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:30.137845 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:47:30.137945 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:30.137953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:30.138193 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:30.138334 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:30.139780 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.140799 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.141957 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.142474 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.142586 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:30.146072 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:30.146082 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:30.146097 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:30.146136 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:30.148675 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.149336 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.149344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.149346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.149463 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.149807 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.149811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.149813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.150243 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:30.150553 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:30.150808 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:47:30.151072 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":209,"threads_started":1}
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:30.168205 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:30.155665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.0.97.62:42945)
0927 22:47:30.155835 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:30.155841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:30.156095 (+ 254us) server_negotiation.cc:407] Connection header received
0927 22:47:30.156247 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:30.156251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:30.156311 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:30.156390 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:30.157513 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.158308 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.159574 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.159979 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.160099 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:30.163581 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:30.163586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:30.163590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:30.163646 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:30.165820 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.166419 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.166423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.166425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.166538 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.166855 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.166859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.166861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.167171 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:30.167388 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:30.167717 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:47:30.167930 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 22:47:30.168711 3995 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51940
I20260927 22:47:30.169582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 4056
I20260927 22:47:30.180859 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:30.170302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.0.97.1:38537)
0927 22:47:30.170513 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:30.170518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:30.170668 (+ 150us) server_negotiation.cc:407] Connection header received
0927 22:47:30.170891 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:30.170897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:30.170973 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:30.171046 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:30.172182 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.172836 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.173921 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.174298 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.174387 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:30.176430 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:30.176434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:30.176438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:30.176467 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:30.178863 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.179399 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.179402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.179405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.179548 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.179834 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.179837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.179839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.180060 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:30.180218 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:30.180408 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:47:30.180556 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:47:30.181365 4179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:30.260011 4184 heartbeater.cc:503] Master 127.0.97.62:42945 was elected leader, sending a full tablet report...
I20260927 22:47:30.260459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 4056
W20260927 22:47:30.273175 3979 connection.cc:535] server connection from 127.0.97.1:54591 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:47:30.273455 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 4190
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:30.287568 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:30.274258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.0.97.2:41445)
0927 22:47:30.274782 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:30.274789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:30.274814 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:47:30.274875 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:30.274881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:30.275082 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:30.275211 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:30.276382 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.277379 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.278531 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.279071 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.279162 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:30.282778 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:30.282788 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:30.282803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:30.282840 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:30.285316 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.285898 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.285905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.285907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.286017 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.286274 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.286277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.286279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.286587 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:30.286803 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:30.286999 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 22:47:30.287176 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":257,"threads_started":1}
I20260927 22:47:30.288838 4313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:30.368237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 4190
I20260927 22:47:30.380550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 4323
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:30.395861 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:30.381700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.0.97.3:41425)
0927 22:47:30.382274 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:30.382281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:30.382308 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:47:30.382363 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:30.382370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:30.382582 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:30.382706 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:30.384003 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.384978 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.386215 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:30.386728 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:30.386819 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:30.390850 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:30.390860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:30.390874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:30.390905 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:30.393511 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.394167 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.394174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.394176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.394288 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:30.394563 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:30.394566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:30.394568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:30.394864 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:30.395107 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:30.395281 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 22:47:30.395500 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260927 22:47:30.397061 4446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:30.478214 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 4323
I20260927 22:47:30.490207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 3963
I20260927 22:47:30.491391 4025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:30.589864 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 3963
2026-09-27T22:47:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3569 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[4470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[4470](info): set up 2 sockets
Sep 27 22:47:30 dist-test-slave-kr25 krb5kdc[4470](info): commencing operation
krb5kdc: starting...
W20260927 22:47:31.579522 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:31 dist-test-slave-kr25 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:31.755182 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34017
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:32.066192 4483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:32.066484 4483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:32.066574 4483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:32.073715 4483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:32.073822 4483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:32.073876 4483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:32.073952 4483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:32.074076 4483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:32.083635 4483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:32.084806 4483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:32.086071 4483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:32.097102 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:32.097174 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:32.097304 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:32.097786 4483 server_base.cc:1034] running on GCE node
I20260927 22:47:32.098493 4483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:32.099879 4483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:32.101071 4483 hybrid_clock.cc:648] HybridClock initialized: now 1790549252101027 us; error 40 us; skew 500 ppm
I20260927 22:47:32.101363 4483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:32 dist-test-slave-kr25 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:32.104192 4483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:32.105213 4483 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:32.105294 4483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:32.105536 4483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:32.108314 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "69d5961979ae4a87bf517e3bd2a3a735"
format_stamp: "Formatted at 2026-09-27 22:47:32 on dist-test-slave-kr25"
server_key: "3cc36967c32a6553b8ceab45ef97af1e"
server_key_iv: "7867cba8a7ad885488576bf0a462d160"
server_key_version: "encryptionkey@0"
I20260927 22:47:32.109004 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "69d5961979ae4a87bf517e3bd2a3a735"
format_stamp: "Formatted at 2026-09-27 22:47:32 on dist-test-slave-kr25"
server_key: "3cc36967c32a6553b8ceab45ef97af1e"
server_key_iv: "7867cba8a7ad885488576bf0a462d160"
server_key_version: "encryptionkey@0"
I20260927 22:47:32.113571 4483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 22:47:32.116659 4497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:32.117853 4483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:32.117983 4483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "69d5961979ae4a87bf517e3bd2a3a735"
format_stamp: "Formatted at 2026-09-27 22:47:32 on dist-test-slave-kr25"
server_key: "3cc36967c32a6553b8ceab45ef97af1e"
server_key_iv: "7867cba8a7ad885488576bf0a462d160"
server_key_version: "encryptionkey@0"
I20260927 22:47:32.118144 4483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:32.145882 4483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:32.149606 4483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:32.149816 4483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:32.165922 4483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34017
I20260927 22:47:32.165902 4548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34017 every 8 connection(s)
I20260927 22:47:32.166797 4483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:32.169672 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 4483
I20260927 22:47:32.169860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:32.170428 388 external_mini_cluster.cc:1428] Setting key 16e9434de9004f7992e4816fc5bd8534
I20260927 22:47:32.170459 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:32.183699 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735: Bootstrap starting.
Sep 27 22:47:32 dist-test-slave-kr25 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:32.186899 4549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:32.187913 4549 log.cc:824] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:32.190701 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735: No bootstrap required, opened a new log
I20260927 22:47:32.191989 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:32.172269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.97.62:34017)
0927 22:47:32.173033 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:32.173048 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:32.173099 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:47:32.175095 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:32.175122 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:32.175654 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:32.176080 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:32.177317 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:32.179223 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:32.180767 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:32.181481 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:32.181664 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:32.185547 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:32.185555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:32.185569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:32.185604 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:32.188378 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:32.189045 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:32.189051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:32.189057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:32.189209 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:32.189555 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:32.189557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:32.189559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:32.190040 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:32.190251 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:32.190636 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 22:47:32.191028 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":310,"threads_started":1}
I20260927 22:47:32.198226 4549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d5961979ae4a87bf517e3bd2a3a735" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34017 } }
I20260927 22:47:32.198527 4549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:32.198616 4549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d5961979ae4a87bf517e3bd2a3a735, State: Initialized, Role: FOLLOWER
I20260927 22:47:32.199100 4549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d5961979ae4a87bf517e3bd2a3a735" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34017 } }
I20260927 22:47:32.199287 4549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:32.199401 4549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:32.199579 4549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:32.201725 4549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d5961979ae4a87bf517e3bd2a3a735" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34017 } }
I20260927 22:47:32.202162 4549 leader_election.cc:302] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d5961979ae4a87bf517e3bd2a3a735; no voters:
I20260927 22:47:32.202641 4549 leader_election.cc:288] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:32.202749 4554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:32.203333 4554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [term 1 LEADER]: Becoming Leader. State: Replica: 69d5961979ae4a87bf517e3bd2a3a735, State: Running, Role: LEADER
I20260927 22:47:32.203825 4554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d5961979ae4a87bf517e3bd2a3a735" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34017 } }
I20260927 22:47:32.204375 4549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:32.206440 4555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d5961979ae4a87bf517e3bd2a3a735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d5961979ae4a87bf517e3bd2a3a735" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34017 } } }
I20260927 22:47:32.206884 4555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:32.207525 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d5961979ae4a87bf517e3bd2a3a735. Latest consensus state: current_term: 1 leader_uuid: "69d5961979ae4a87bf517e3bd2a3a735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d5961979ae4a87bf517e3bd2a3a735" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34017 } } }
I20260927 22:47:32.207832 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:32.209004 4559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:32.214053 4559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:32.222061 4559 catalog_manager.cc:1347] Generated new cluster ID: a61a520fc30a4e78a17ee1287127c5d7
I20260927 22:47:32.222175 4559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:32.258787 4559 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:32.260259 4559 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:32.269853 4559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735: Generated new TSK 0
I20260927 22:47:32.270890 4559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:32.324966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34017
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:32.667526 4575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:32.667889 4575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:32.668041 4575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:32.676409 4575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:32.676509 4575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:32.676630 4575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:32.686080 4575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:32.687282 4575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:32.688549 4575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:32.700333 4580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:32.700371 4583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:32.700397 4581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:32.701367 4575 server_base.cc:1034] running on GCE node
I20260927 22:47:32.701930 4575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:32.702776 4575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:32.703971 4575 hybrid_clock.cc:648] HybridClock initialized: now 1790549252703927 us; error 32 us; skew 500 ppm
I20260927 22:47:32.704276 4575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:32 dist-test-slave-kr25 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:32.707088 4575 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:32.708190 4575 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:32.708293 4575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:32.708557 4575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:32.711395 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "bca57691eeb14b07a0247c8041e5d184"
format_stamp: "Formatted at 2026-09-27 22:47:32 on dist-test-slave-kr25"
server_key: "0ab1159e422c98f2cbe32c892bcda2de"
server_key_iv: "c9cd553927902ed4346df87c18588a2c"
server_key_version: "encryptionkey@0"
I20260927 22:47:32.712145 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bca57691eeb14b07a0247c8041e5d184"
format_stamp: "Formatted at 2026-09-27 22:47:32 on dist-test-slave-kr25"
server_key: "0ab1159e422c98f2cbe32c892bcda2de"
server_key_iv: "c9cd553927902ed4346df87c18588a2c"
server_key_version: "encryptionkey@0"
I20260927 22:47:32.716658 4575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 22:47:32.719622 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:32.720775 4575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:47:32.720897 4575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "bca57691eeb14b07a0247c8041e5d184"
format_stamp: "Formatted at 2026-09-27 22:47:32 on dist-test-slave-kr25"
server_key: "0ab1159e422c98f2cbe32c892bcda2de"
server_key_iv: "c9cd553927902ed4346df87c18588a2c"
server_key_version: "encryptionkey@0"
I20260927 22:47:32.721076 4575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:32.738451 4575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:32.742097 4575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:32.742329 4575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:32.743194 4575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:32.744477 4575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:32.744557 4575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:32.744668 4575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:32.744740 4575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:32.772356 4575 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37555
I20260927 22:47:32.772439 4701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37555 every 8 connection(s)
I20260927 22:47:32.773144 4575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:32.774991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 4575
I20260927 22:47:32.775180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:32.775882 388 external_mini_cluster.cc:1428] Setting key 209b3fb46806b2d8e1c906a301e788f4
Sep 27 22:47:32 dist-test-slave-kr25 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:32.791383 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:32.775560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44933 (local address 127.0.97.62:34017)
0927 22:47:32.776007 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:32.776014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:32.777010 (+ 996us) server_negotiation.cc:407] Connection header received
0927 22:47:32.778737 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:32.778743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:32.778819 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:32.778910 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:32.780380 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:32.781069 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:32.782544 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:32.782927 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:32.783033 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:32.786353 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:32.786358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:32.786361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:32.786389 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:32.788689 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:32.789325 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:32.789330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:32.789333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:32.789444 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:32.789805 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:32.789810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:32.789813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:32.790064 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:32.790186 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:32.790831 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 22:47:32.791039 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":205,"threads_started":1}
I20260927 22:47:32.792325 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:32.776142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34017 (local address 127.0.97.1:44933)
0927 22:47:32.776865 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:32.776907 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:32.778487 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:32.779142 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:32.779150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:32.779679 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:32.780220 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:32.780240 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:32.781208 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:32.781212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:32.782409 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:32.782420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:32.783138 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:32.783145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:32.783355 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:32.783989 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:32.784011 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:32.786164 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:32.788806 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:32.788815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:32.788828 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:32.789213 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:32.789535 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:32.789539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:32.789541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:32.789698 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:32.790175 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:32.790182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:32.790482 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 22:47:32.790704 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 22:47:32.791208 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:47:32.793918 4702 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34017
I20260927 22:47:32.794194 4702 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:32.794813 4702 heartbeater.cc:511] Master 127.0.97.62:34017 requested a full tablet report, sending...
I20260927 22:47:32.796967 4514 ts_manager.cc:194] Registered new tserver with Master: bca57691eeb14b07a0247c8041e5d184 (127.0.97.1:37555)
I20260927 22:47:32.798601 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:32.814937 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34017
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:33.119678 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:33.119958 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:33.120091 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:33.127409 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:33.127562 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:33.127712 4708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:33.137359 4708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:33.138684 4708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:33.140033 4708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:33.152396 4713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:33.152421 4714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:33.152375 4716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:33.153339 4708 server_base.cc:1034] running on GCE node
I20260927 22:47:33.153923 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:33.154850 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:33.156052 4708 hybrid_clock.cc:648] HybridClock initialized: now 1790549253155988 us; error 54 us; skew 500 ppm
I20260927 22:47:33.156368 4708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:33 dist-test-slave-kr25 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:33.159340 4708 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:33.160494 4708 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:33.160597 4708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:33.160912 4708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:33.163812 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "23b316251d204ea48dd082aa9c0c9cec"
format_stamp: "Formatted at 2026-09-27 22:47:33 on dist-test-slave-kr25"
server_key: "9973313c1d5d7ded8d5d4b0b302a4f2d"
server_key_iv: "f4af359af16b8c3490aec0feee924db1"
server_key_version: "encryptionkey@0"
I20260927 22:47:33.164595 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "23b316251d204ea48dd082aa9c0c9cec"
format_stamp: "Formatted at 2026-09-27 22:47:33 on dist-test-slave-kr25"
server_key: "9973313c1d5d7ded8d5d4b0b302a4f2d"
server_key_iv: "f4af359af16b8c3490aec0feee924db1"
server_key_version: "encryptionkey@0"
I20260927 22:47:33.169687 4708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:47:33.173254 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:33.174773 4708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:47:33.174933 4708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "23b316251d204ea48dd082aa9c0c9cec"
format_stamp: "Formatted at 2026-09-27 22:47:33 on dist-test-slave-kr25"
server_key: "9973313c1d5d7ded8d5d4b0b302a4f2d"
server_key_iv: "f4af359af16b8c3490aec0feee924db1"
server_key_version: "encryptionkey@0"
I20260927 22:47:33.175134 4708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:33.227212 4708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:33.231324 4708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:33.231599 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:33.232604 4708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:33.233940 4708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:33.234033 4708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:33.234141 4708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:33.234207 4708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:33.263195 4708 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36319
I20260927 22:47:33.263240 4834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36319 every 8 connection(s)
I20260927 22:47:33.264024 4708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:33.271801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 4708
I20260927 22:47:33.271989 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:33.272531 388 external_mini_cluster.cc:1428] Setting key b3591b16377757c7a77761211a006507
Sep 27 22:47:33 dist-test-slave-kr25 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:33.283087 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.266259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49729 (local address 127.0.97.62:34017)
0927 22:47:33.266449 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:33.266455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:33.267505 (+ 1050us) server_negotiation.cc:407] Connection header received
0927 22:47:33.269195 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:33.269200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:33.269276 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:33.269362 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:33.270683 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.271339 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.273393 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.273869 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.273961 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.277519 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:33.277523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:33.277527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:33.277556 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:33.280320 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:33.280972 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:33.280977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:33.280981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:33.281128 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:33.281489 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:33.281493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:33.281497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:33.281804 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:33.281932 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:33.282548 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 22:47:33.282755 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260927 22:47:33.283941 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.266702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34017 (local address 127.0.97.2:49729)
0927 22:47:33.267339 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:33.267379 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:33.268941 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:33.269532 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:33.269540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:33.270007 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:33.270540 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.270555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.271503 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.271509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.273280 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.273291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.274077 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.274084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.274338 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.275037 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:33.275058 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:33.277309 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:33.280434 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:33.280440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:33.280453 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:33.280859 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:33.281220 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:33.281223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:33.281225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:33.281381 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:33.281904 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:33.281910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:33.282220 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 22:47:33.282430 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 22:47:33.282806 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":237,"threads_started":1}
I20260927 22:47:33.285341 4835 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34017
I20260927 22:47:33.285678 4835 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:33.286433 4835 heartbeater.cc:511] Master 127.0.97.62:34017 requested a full tablet report, sending...
I20260927 22:47:33.288005 4514 ts_manager.cc:194] Registered new tserver with Master: 23b316251d204ea48dd082aa9c0c9cec (127.0.97.2:36319)
I20260927 22:47:33.288759 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49729
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:33.313766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34017
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:33.619238 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:33.619570 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:33.619690 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:33.626928 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:33.627038 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:33.627159 4840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:33.636776 4840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:33.638015 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:33.639263 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:33.651171 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:33.651239 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:33.651170 4846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:33.652027 4840 server_base.cc:1034] running on GCE node
I20260927 22:47:33.652602 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:33.653524 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:33.654727 4840 hybrid_clock.cc:648] HybridClock initialized: now 1790549253654677 us; error 33 us; skew 500 ppm
I20260927 22:47:33.655021 4840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:33 dist-test-slave-kr25 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:33.657964 4840 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:33.659054 4840 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:33.659155 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:33.659492 4840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:33.662405 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "d4c17902336e45159fbd29e54e1bd5d2"
format_stamp: "Formatted at 2026-09-27 22:47:33 on dist-test-slave-kr25"
server_key: "cf3d6defef0f46832133cbe98416f66d"
server_key_iv: "1747fd997a360caaa75443cfa6265f7e"
server_key_version: "encryptionkey@0"
I20260927 22:47:33.663233 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d4c17902336e45159fbd29e54e1bd5d2"
format_stamp: "Formatted at 2026-09-27 22:47:33 on dist-test-slave-kr25"
server_key: "cf3d6defef0f46832133cbe98416f66d"
server_key_iv: "1747fd997a360caaa75443cfa6265f7e"
server_key_version: "encryptionkey@0"
I20260927 22:47:33.668242 4840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:47:33.671610 4854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:33.672998 4840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:47:33.673153 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "d4c17902336e45159fbd29e54e1bd5d2"
format_stamp: "Formatted at 2026-09-27 22:47:33 on dist-test-slave-kr25"
server_key: "cf3d6defef0f46832133cbe98416f66d"
server_key_iv: "1747fd997a360caaa75443cfa6265f7e"
server_key_version: "encryptionkey@0"
I20260927 22:47:33.673379 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:33.695372 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:33.699630 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:33.699898 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:33.700888 4840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:33.702265 4840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:33.702365 4840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:33.702491 4840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:33.702567 4840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:33.733567 4840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44553
I20260927 22:47:33.733615 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44553 every 8 connection(s)
I20260927 22:47:33.734413 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:33.739269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 4840
I20260927 22:47:33.739459 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:33.739866 388 external_mini_cluster.cc:1428] Setting key e51747c5c5256ca90b19e1c3ae3cdc47
Sep 27 22:47:33 dist-test-slave-kr25 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:33.752431 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.736734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60877 (local address 127.0.97.62:34017)
0927 22:47:33.736916 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:33.736921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:33.738085 (+ 1164us) server_negotiation.cc:407] Connection header received
0927 22:47:33.740049 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:33.740056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:33.740142 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:33.740237 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:33.741912 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.742758 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.744278 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.744652 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.744746 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.747856 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:33.747861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:33.747864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:33.747893 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:33.750001 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:33.750609 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:33.750612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:33.750615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:33.750721 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:33.751051 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:33.751053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:33.751055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:33.751299 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:33.751404 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:33.752014 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 22:47:33.752178 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:47:33.753482 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.737226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34017 (local address 127.0.97.3:60877)
0927 22:47:33.737913 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:33.737951 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:33.739786 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:33.740474 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:33.740485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:33.741086 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:33.741697 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.741725 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.742887 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.742891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.744150 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.744163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.744820 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.744825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.745011 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.745582 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:33.745603 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:33.747682 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:33.750117 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:33.750123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:33.750134 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:33.750519 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:33.750798 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:33.750801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:33.750803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:33.750958 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:33.751390 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:33.751395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:33.751709 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 22:47:33.751922 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 22:47:33.752289 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260927 22:47:33.754889 4967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34017
I20260927 22:47:33.755190 4967 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:33.755827 4967 heartbeater.cc:511] Master 127.0.97.62:34017 requested a full tablet report, sending...
I20260927 22:47:33.757155 4514 ts_manager.cc:194] Registered new tserver with Master: d4c17902336e45159fbd29e54e1bd5d2 (127.0.97.3:44553)
I20260927 22:47:33.757927 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60877
I20260927 22:47:33.766546 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:47:33.783221 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.770891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.97.62:34017)
0927 22:47:33.771089 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:33.771095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:33.771474 (+ 379us) server_negotiation.cc:407] Connection header received
0927 22:47:33.771629 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:33.771633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:33.771693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:33.771783 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:33.773151 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.773841 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.775046 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.775484 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.775590 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.778142 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:33.778147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:33.778150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:33.778183 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:33.780624 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:33.781259 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:33.781266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:33.781268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:33.781385 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:33.781757 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:33.781763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:33.781767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:33.782100 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:33.782331 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:33.782692 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 22:47:33.782909 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:47:33.786934 4514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:33.793442 4514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:33.801769 4702 heartbeater.cc:503] Master 127.0.97.62:34017 was elected leader, sending a full tablet report...
I20260927 22:47:33.815200 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.804319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37555 (local address 127.0.0.1:54660)
0927 22:47:33.805780 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:33.805818 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:33.805961 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:33.806891 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:33.806899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:33.806928 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:33.807276 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.807288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.809393 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.809399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.811942 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.811958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.814202 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.814209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.814427 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.814478 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:47:33.814617 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:47:33.814806 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":1145,"threads_started":1}
I20260927 22:47:33.816135 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.804634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.0.97.1:37555)
0927 22:47:33.805478 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:33.805486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:33.805909 (+ 423us) server_negotiation.cc:407] Connection header received
0927 22:47:33.806143 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:33.806151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:33.806412 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:33.806610 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:33.807487 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.809182 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.812135 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.813687 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.813820 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.814063 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:33.815613 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0927 22:47:33.815802 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":500,"threads_started":1}
I20260927 22:47:33.821527 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf3ee07aafe485fbcf065f15c93ea15 (DEFAULT_TABLE table=test-table [id=f53a1799725442f09f0182cffc9e5613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:33.823035 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf3ee07aafe485fbcf065f15c93ea15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:33.824584 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.806147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36319 (local address 127.0.0.1:56568)
0927 22:47:33.806619 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:33.806641 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:33.806769 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:33.818785 (+ 12016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:33.818789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:33.818811 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:33.819076 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.819085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.820758 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.820763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.822502 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.822513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.823824 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.823830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.823983 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.824013 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:47:33.824098 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:47:33.824268 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260927 22:47:33.824594 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.807222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.0.97.3:44553)
0927 22:47:33.816988 (+ 9766us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:33.816994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:33.817020 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:47:33.817098 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:33.817104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:33.817292 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:33.817435 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:33.818117 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.819411 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.821664 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.823052 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.823173 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.823392 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:33.824083 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 22:47:33.824280 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9632,"thread_start_us":9510,"threads_started":1}
I20260927 22:47:33.824806 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.806351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.0.97.2:36319)
0927 22:47:33.818268 (+ 11917us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:33.818276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:33.818300 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:47:33.818387 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:33.818394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:33.818604 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:33.818783 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:33.819204 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.820616 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.822662 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.823630 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.823738 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.823907 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:33.824410 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 22:47:33.824550 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11785,"thread_start_us":11692,"threads_started":1}
I20260927 22:47:33.824294 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:33.806685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44553 (local address 127.0.0.1:57566)
0927 22:47:33.816402 (+ 9717us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:33.816420 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:33.816522 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:33.817470 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:33.817475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:33.817504 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:33.817897 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.817908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.819551 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.819556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.821506 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:33.821519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:33.823534 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:33.823540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:33.823694 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:33.823739 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:47:33.823832 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 22:47:33.823951 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9609,"thread_start_us":10905,"threads_started":1}
I20260927 22:47:33.827991 4770 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf3ee07aafe485fbcf065f15c93ea15 (DEFAULT_TABLE table=test-table [id=f53a1799725442f09f0182cffc9e5613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:33.828042 4902 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf3ee07aafe485fbcf065f15c93ea15 (DEFAULT_TABLE table=test-table [id=f53a1799725442f09f0182cffc9e5613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:33.829021 4770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf3ee07aafe485fbcf065f15c93ea15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:33.829022 4902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf3ee07aafe485fbcf065f15c93ea15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:33.839030 4985 tablet_bootstrap.cc:492] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Bootstrap starting.
I20260927 22:47:33.842100 4985 tablet_bootstrap.cc:654] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:33.843101 4985 log.cc:824] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:33.845192 4988 tablet_bootstrap.cc:492] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Bootstrap starting.
I20260927 22:47:33.845584 4986 tablet_bootstrap.cc:492] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Bootstrap starting.
I20260927 22:47:33.846447 4985 tablet_bootstrap.cc:492] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: No bootstrap required, opened a new log
I20260927 22:47:33.846668 4985 ts_tablet_manager.cc:1397] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:47:33.848491 4988 tablet_bootstrap.cc:654] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:33.848629 4986 tablet_bootstrap.cc:654] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:33.849642 4986 log.cc:824] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:33.849637 4988 log.cc:824] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:33.852262 4988 tablet_bootstrap.cc:492] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: No bootstrap required, opened a new log
I20260927 22:47:33.852507 4988 ts_tablet_manager.cc:1397] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:47:33.852734 4986 tablet_bootstrap.cc:492] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: No bootstrap required, opened a new log
I20260927 22:47:33.853017 4986 ts_tablet_manager.cc:1397] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:47:33.854663 4985 raft_consensus.cc:348] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:33.855083 4985 raft_consensus.cc:374] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:33.855181 4985 raft_consensus.cc:729] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b316251d204ea48dd082aa9c0c9cec, State: Initialized, Role: FOLLOWER
I20260927 22:47:33.855787 4985 consensus_queue.cc:260] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:33.856825 4835 heartbeater.cc:503] Master 127.0.97.62:34017 was elected leader, sending a full tablet report...
I20260927 22:47:33.857419 4985 ts_tablet_manager.cc:1428] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260927 22:47:33.859977 4988 raft_consensus.cc:348] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:33.860296 4988 raft_consensus.cc:374] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:33.860384 4988 raft_consensus.cc:729] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca57691eeb14b07a0247c8041e5d184, State: Initialized, Role: FOLLOWER
I20260927 22:47:33.861003 4988 consensus_queue.cc:260] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:33.862104 4986 raft_consensus.cc:348] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:33.862498 4986 raft_consensus.cc:374] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:33.862582 4986 raft_consensus.cc:729] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c17902336e45159fbd29e54e1bd5d2, State: Initialized, Role: FOLLOWER
I20260927 22:47:33.862643 4988 ts_tablet_manager.cc:1428] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 22:47:33.863101 4986 consensus_queue.cc:260] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:33.864059 4967 heartbeater.cc:503] Master 127.0.97.62:34017 was elected leader, sending a full tablet report...
I20260927 22:47:33.864527 4986 ts_tablet_manager.cc:1428] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
W20260927 22:47:33.985864 4968 tablet.cc:2369] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:47:34.015712 4836 tablet.cc:2369] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:47:34.025055 4703 tablet.cc:2369] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:34.131006 4993 raft_consensus.cc:482] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:34.131289 4993 raft_consensus.cc:504] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:34.132963 4993 leader_election.cc:288] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bca57691eeb14b07a0247c8041e5d184 (127.0.97.1:37555), 23b316251d204ea48dd082aa9c0c9cec (127.0.97.2:36319)
I20260927 22:47:34.139065 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.133197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37555 (local address 127.0.97.3:50987)
0927 22:47:34.133403 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:34.133421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:34.133565 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:34.134015 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:34.134019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:34.134052 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:34.134236 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:34.134248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.135487 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.135492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:34.137285 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:34.137300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.138552 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.138555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:34.138683 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.138706 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:47:34.138781 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:47:34.138870 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:47:34.139190 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.133395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50987 (local address 127.0.97.1:37555)
0927 22:47:34.133588 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:34.133593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:34.133614 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:47:34.133684 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:34.133687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:34.133757 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:34.133836 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:34.134388 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.135320 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.137444 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.138404 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.138538 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.138638 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:34.138826 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 22:47:34.138950 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:47:34.140071 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.133835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41307 (local address 127.0.97.2:36319)
0927 22:47:34.134013 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:34.134019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:34.134339 (+ 320us) server_negotiation.cc:407] Connection header received
0927 22:47:34.134482 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:34.134487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:34.134560 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:34.134667 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:34.135082 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.135870 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.137894 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.139018 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.139142 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.139226 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:34.139777 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 22:47:34.139898 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:47:34.140139 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.133616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36319 (local address 127.0.97.3:41307)
0927 22:47:34.134248 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:34.134266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:34.134372 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:34.134670 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:34.134673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:34.134689 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:34.134959 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:34.134966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.136014 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.136019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:34.137761 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:34.137771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.139359 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.139365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:34.139567 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.139600 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:47:34.139729 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:47:34.139848 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":292,"threads_started":1}
I20260927 22:47:34.140357 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf3ee07aafe485fbcf065f15c93ea15" candidate_uuid: "d4c17902336e45159fbd29e54e1bd5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca57691eeb14b07a0247c8041e5d184" is_pre_election: true
I20260927 22:47:34.140882 4657 raft_consensus.cc:2454] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4c17902336e45159fbd29e54e1bd5d2 in term 0.
I20260927 22:47:34.141063 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf3ee07aafe485fbcf065f15c93ea15" candidate_uuid: "d4c17902336e45159fbd29e54e1bd5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b316251d204ea48dd082aa9c0c9cec" is_pre_election: true
I20260927 22:47:34.141501 4855 leader_election.cc:302] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bca57691eeb14b07a0247c8041e5d184, d4c17902336e45159fbd29e54e1bd5d2; no voters:
I20260927 22:47:34.141572 4790 raft_consensus.cc:2454] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4c17902336e45159fbd29e54e1bd5d2 in term 0.
I20260927 22:47:34.141861 4993 raft_consensus.cc:2781] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:34.142025 4993 raft_consensus.cc:482] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:34.142127 4993 raft_consensus.cc:3037] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:34.144301 4993 raft_consensus.cc:504] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:34.144814 4993 leader_election.cc:288] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [CANDIDATE]: Term 1 election: Requested vote from peers bca57691eeb14b07a0247c8041e5d184 (127.0.97.1:37555), 23b316251d204ea48dd082aa9c0c9cec (127.0.97.2:36319)
I20260927 22:47:34.145243 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf3ee07aafe485fbcf065f15c93ea15" candidate_uuid: "d4c17902336e45159fbd29e54e1bd5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca57691eeb14b07a0247c8041e5d184"
I20260927 22:47:34.145279 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf3ee07aafe485fbcf065f15c93ea15" candidate_uuid: "d4c17902336e45159fbd29e54e1bd5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b316251d204ea48dd082aa9c0c9cec"
I20260927 22:47:34.145403 4657 raft_consensus.cc:3037] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:34.145444 4790 raft_consensus.cc:3037] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:34.147389 4657 raft_consensus.cc:2454] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4c17902336e45159fbd29e54e1bd5d2 in term 1.
I20260927 22:47:34.147455 4790 raft_consensus.cc:2454] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4c17902336e45159fbd29e54e1bd5d2 in term 1.
I20260927 22:47:34.147821 4855 leader_election.cc:302] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b316251d204ea48dd082aa9c0c9cec, d4c17902336e45159fbd29e54e1bd5d2; no voters:
I20260927 22:47:34.148049 4993 raft_consensus.cc:2781] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:34.148454 4993 raft_consensus.cc:686] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 1 LEADER]: Becoming Leader. State: Replica: d4c17902336e45159fbd29e54e1bd5d2, State: Running, Role: LEADER
I20260927 22:47:34.148926 4993 consensus_queue.cc:237] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } }
I20260927 22:47:34.153359 4512 catalog_manager.cc:5495] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4c17902336e45159fbd29e54e1bd5d2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d4c17902336e45159fbd29e54e1bd5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4c17902336e45159fbd29e54e1bd5d2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44553 } health_report { overall_health: HEALTHY } } }
I20260927 22:47:34.177870 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.172181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.0.97.3:44553)
0927 22:47:34.172328 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:34.172332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:34.172348 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:47:34.172426 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:34.172429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:34.172507 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:34.172626 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:34.172972 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.173686 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.175633 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.176103 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.176197 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.176790 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:34.177318 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:34.177459 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:34.177562 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 22:47:34.177640 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:47:34.185434 4657 raft_consensus.cc:1261] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 1 FOLLOWER]: Refusing update from remote peer d4c17902336e45159fbd29e54e1bd5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:34.185431 4790 raft_consensus.cc:1261] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 1 FOLLOWER]: Refusing update from remote peer d4c17902336e45159fbd29e54e1bd5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:34.186266 4995 consensus_queue.cc:1035] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b316251d204ea48dd082aa9c0c9cec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:34.186560 4993 consensus_queue.cc:1035] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bca57691eeb14b07a0247c8041e5d184" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:34.196753 4998 mvcc.cc:205] Tried to move back new op lower bound from 7334089745134313472 to 7334089744997302272. Current Snapshot: MvccSnapshot[applied={T|T < 7334089745134313472}]
I20260927 22:47:34.209889 4512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:34.210175 4512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:34.213503 4512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69d5961979ae4a87bf517e3bd2a3a735: Sending DeleteTablet for 3 replicas of tablet 5bf3ee07aafe485fbcf065f15c93ea15
I20260927 22:47:34.214665 4770 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf3ee07aafe485fbcf065f15c93ea15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56568
I20260927 22:47:34.214771 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf3ee07aafe485fbcf065f15c93ea15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54660
I20260927 22:47:34.214996 4902 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf3ee07aafe485fbcf065f15c93ea15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57566
I20260927 22:47:34.216420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 4575
I20260927 22:47:34.227306 5009 tablet_replica.cc:330] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: stopping tablet replica
I20260927 22:47:34.227938 5009 raft_consensus.cc:2229] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:34.228539 5009 raft_consensus.cc:2258] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:34.229292 5008 tablet_replica.cc:330] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: stopping tablet replica
I20260927 22:47:34.229589 5008 raft_consensus.cc:2229] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:34.229962 5008 raft_consensus.cc:2258] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:34.231608 5009 ts_tablet_manager.cc:1905] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:34.237313 5008 ts_tablet_manager.cc:1905] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:34.237330 5009 ts_tablet_manager.cc:1918] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:34.237459 5009 log.cc:1196] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/5bf3ee07aafe485fbcf065f15c93ea15
I20260927 22:47:34.237958 5009 ts_tablet_manager.cc:1939] T 5bf3ee07aafe485fbcf065f15c93ea15 P bca57691eeb14b07a0247c8041e5d184: Deleting consensus metadata
I20260927 22:47:34.239615 4498 catalog_manager.cc:4841] TS bca57691eeb14b07a0247c8041e5d184 (127.0.97.1:37555): tablet 5bf3ee07aafe485fbcf065f15c93ea15 (table test-table [id=f53a1799725442f09f0182cffc9e5613]) successfully deleted
I20260927 22:47:34.241931 5010 tablet_replica.cc:330] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: stopping tablet replica
I20260927 22:47:34.242322 5010 raft_consensus.cc:2229] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:34.243027 5010 raft_consensus.cc:2258] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:34.243556 5008 ts_tablet_manager.cc:1918] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:34.243747 5008 log.cc:1196] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/5bf3ee07aafe485fbcf065f15c93ea15
I20260927 22:47:34.244298 5008 ts_tablet_manager.cc:1939] T 5bf3ee07aafe485fbcf065f15c93ea15 P 23b316251d204ea48dd082aa9c0c9cec: Deleting consensus metadata
I20260927 22:47:34.245083 5010 ts_tablet_manager.cc:1905] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:34.245963 4498 catalog_manager.cc:4841] TS 23b316251d204ea48dd082aa9c0c9cec (127.0.97.2:36319): tablet 5bf3ee07aafe485fbcf065f15c93ea15 (table test-table [id=f53a1799725442f09f0182cffc9e5613]) successfully deleted
I20260927 22:47:34.250505 5010 ts_tablet_manager.cc:1918] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:34.250638 5010 log.cc:1196] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/5bf3ee07aafe485fbcf065f15c93ea15
I20260927 22:47:34.251057 5010 ts_tablet_manager.cc:1939] T 5bf3ee07aafe485fbcf065f15c93ea15 P d4c17902336e45159fbd29e54e1bd5d2: Deleting consensus metadata
I20260927 22:47:34.252404 4498 catalog_manager.cc:4841] TS d4c17902336e45159fbd29e54e1bd5d2 (127.0.97.3:44553): tablet 5bf3ee07aafe485fbcf065f15c93ea15 (table test-table [id=f53a1799725442f09f0182cffc9e5613]) successfully deleted
Sep 27 22:47:34 dist-test-slave-kr25 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:34.262564 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.217804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.97.1:37555)
0927 22:47:34.217958 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:34.217964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:34.241237 (+ 23273us) server_negotiation.cc:407] Connection header received
0927 22:47:34.241448 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:34.241453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:34.241534 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:34.241603 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:34.243251 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.244359 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.246123 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.246742 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.246834 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.256351 (+ 9517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:34.256367 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:34.256386 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:34.256437 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:34.259937 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:34.260564 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:34.260573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:34.260577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:34.260729 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:34.261052 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:34.261057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:34.261061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:34.261510 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:34.261812 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:34.261978 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 22:47:34.262200 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:47:34.263249 4697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:34.373943 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 4575
I20260927 22:47:34.387985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 4708
Sep 27 22:47:34 dist-test-slave-kr25 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:34.403503 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.388695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.0.97.2:36319)
0927 22:47:34.388843 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:34.388847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:34.388869 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:47:34.389055 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:34.389059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:34.389129 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:34.389208 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:34.390665 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.391351 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.393081 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.393491 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.393582 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.398316 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:34.398330 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:34.398346 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:34.398378 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:34.400999 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:34.401707 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:34.401713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:34.401716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:34.401834 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:34.402189 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:34.402191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:34.402193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:34.402520 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:34.402714 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:34.403011 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:47:34.403157 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:47:34.404157 4830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:34.536453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 4708
I20260927 22:47:34.549335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 4840
Sep 27 22:47:34 dist-test-slave-kr25 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:34.565517 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:34.550034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.0.97.3:44553)
0927 22:47:34.550203 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:34.550207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:34.550269 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:47:34.550443 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:34.550447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:34.550512 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:34.550646 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:34.552196 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.552968 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.554726 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:34.555158 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:34.555246 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:34.560025 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:34.560037 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:34.560051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:34.560088 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:34.562860 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:34.563612 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:34.563624 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:34.563627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:34.563777 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:34.564130 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:34.564136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:34.564139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:34.564449 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:34.564793 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:34.564989 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 22:47:34.565187 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:47:34.566192 4962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:34.684412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 4840
I20260927 22:47:34.699473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 4483
I20260927 22:47:34.700467 4544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:34.803283 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 4483
2026-09-27T22:47:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4212 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T22:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:34Z Disabled control of system clock
I20260927 22:47:34.859714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41579
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:47:35.162086 5019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:35.162380 5019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:35.162465 5019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:35.169498 5019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:35.169608 5019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:35.169646 5019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:35.169695 5019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:35.169739 5019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:35.178762 5019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41579
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:35.180110 5019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:35.181788 5019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:35.192490 5024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:35.192572 5027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:35.192531 5025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:35.193475 5019 server_base.cc:1034] running on GCE node
I20260927 22:47:35.194180 5019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:35.195523 5019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:35.196714 5019 hybrid_clock.cc:648] HybridClock initialized: now 1790549255196668 us; error 41 us; skew 500 ppm
I20260927 22:47:35.197028 5019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:35.199061 5019 webserver.cc:460] Webserver started at http://127.0.97.62:37071/ using document root <none> and password file <none>
I20260927 22:47:35.199750 5019 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:35.199848 5019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:35.200146 5019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:35.202845 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "dbdcdae86596410eba784419496604b1"
format_stamp: "Formatted at 2026-09-27 22:47:35 on dist-test-slave-kr25"
server_key: "bcd4a1c773343e5444301c31b9469cdd"
server_key_iv: "77dcc39cd47c10594ca0686e70e6dc15"
server_key_version: "encryptionkey@0"
I20260927 22:47:35.203608 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "dbdcdae86596410eba784419496604b1"
format_stamp: "Formatted at 2026-09-27 22:47:35 on dist-test-slave-kr25"
server_key: "bcd4a1c773343e5444301c31b9469cdd"
server_key_iv: "77dcc39cd47c10594ca0686e70e6dc15"
server_key_version: "encryptionkey@0"
I20260927 22:47:35.208334 5019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:47:35.211390 5033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:35.212652 5019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 22:47:35.212802 5019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "dbdcdae86596410eba784419496604b1"
format_stamp: "Formatted at 2026-09-27 22:47:35 on dist-test-slave-kr25"
server_key: "bcd4a1c773343e5444301c31b9469cdd"
server_key_iv: "77dcc39cd47c10594ca0686e70e6dc15"
server_key_version: "encryptionkey@0"
I20260927 22:47:35.212986 5019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:35.218354 5019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:35.219051 5019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:35.219316 5019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:35.235735 5019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41579
I20260927 22:47:35.235764 5084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41579 every 8 connection(s)
I20260927 22:47:35.236975 5019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:35.240517 5085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:35.244421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5019
I20260927 22:47:35.244656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:35.245299 388 external_mini_cluster.cc:1428] Setting key 96fe8bed591e147e6e1a361b936cb6f7
I20260927 22:47:35.250674 5085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1: Bootstrap starting.
I20260927 22:47:35.253577 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:35.247244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.97.62:41579)
0927 22:47:35.248282 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:35.248292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:35.248345 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:47:35.250033 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:35.250056 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:35.250463 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:35.250969 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:35.252042 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:35.252048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:35.252061 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:35.252076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:35.252144 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:35.252163 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:35.252344 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:35.252435 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 22:47:35.252658 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":389,"threads_started":1}
I20260927 22:47:35.254173 5085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:35.255292 5085 log.cc:824] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:35.257870 5085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1: No bootstrap required, opened a new log
I20260927 22:47:35.264619 5085 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdcdae86596410eba784419496604b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41579 } }
I20260927 22:47:35.264935 5085 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:35.265025 5085 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdcdae86596410eba784419496604b1, State: Initialized, Role: FOLLOWER
I20260927 22:47:35.265524 5085 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdcdae86596410eba784419496604b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41579 } }
I20260927 22:47:35.265723 5085 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:35.265836 5085 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:35.265966 5085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:35.267993 5085 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdcdae86596410eba784419496604b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41579 } }
I20260927 22:47:35.268522 5085 leader_election.cc:302] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbdcdae86596410eba784419496604b1; no voters:
I20260927 22:47:35.269037 5085 leader_election.cc:288] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:35.269147 5090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:35.269742 5090 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [term 1 LEADER]: Becoming Leader. State: Replica: dbdcdae86596410eba784419496604b1, State: Running, Role: LEADER
I20260927 22:47:35.270258 5090 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdcdae86596410eba784419496604b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41579 } }
I20260927 22:47:35.270884 5085 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:35.273434 5091 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbdcdae86596410eba784419496604b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdcdae86596410eba784419496604b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41579 } } }
I20260927 22:47:35.273394 5092 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbdcdae86596410eba784419496604b1. Latest consensus state: current_term: 1 leader_uuid: "dbdcdae86596410eba784419496604b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdcdae86596410eba784419496604b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41579 } } }
I20260927 22:47:35.273772 5092 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:35.273768 5091 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:35.274598 5099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:35.279639 5099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:35.288549 5099 catalog_manager.cc:1347] Generated new cluster ID: 112120f834634433932689bdb51ccea5
I20260927 22:47:35.288669 5099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:35.306643 5099 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:35.307077 5099 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:35.318871 5099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1: Generated new TSK 0
I20260927 22:47:35.319758 5099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:47:35.325120 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41579
--builtin_ntp_servers=127.0.97.20:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:47:35.650017 5109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:35.650278 5109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:35.650364 5109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:35.657473 5109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:35.657578 5109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:35.657691 5109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:35.666607 5109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:35.667819 5109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:35.669448 5109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:35.683944 5114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:35.683917 5117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:35.683873 5115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:35.684530 5109 server_base.cc:1034] running on GCE node
I20260927 22:47:35.685072 5109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:35.685930 5109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:35.687114 5109 hybrid_clock.cc:648] HybridClock initialized: now 1790549255687067 us; error 43 us; skew 500 ppm
I20260927 22:47:35.687397 5109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:35.689415 5109 webserver.cc:460] Webserver started at http://127.0.97.1:45025/ using document root <none> and password file <none>
I20260927 22:47:35.690040 5109 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:35.690130 5109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:35.690409 5109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:35.693113 5109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "6f992b0fcdea4981b05faa152af4edcc"
format_stamp: "Formatted at 2026-09-27 22:47:35 on dist-test-slave-kr25"
server_key: "2104669f8b71560ddc198939a23191e5"
server_key_iv: "37c2ee9465b73c7bd601cbd755fd3509"
server_key_version: "encryptionkey@0"
I20260927 22:47:35.693897 5109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6f992b0fcdea4981b05faa152af4edcc"
format_stamp: "Formatted at 2026-09-27 22:47:35 on dist-test-slave-kr25"
server_key: "2104669f8b71560ddc198939a23191e5"
server_key_iv: "37c2ee9465b73c7bd601cbd755fd3509"
server_key_version: "encryptionkey@0"
I20260927 22:47:35.698566 5109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:47:35.701695 5123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:35.702888 5109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:35.703030 5109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "6f992b0fcdea4981b05faa152af4edcc"
format_stamp: "Formatted at 2026-09-27 22:47:35 on dist-test-slave-kr25"
server_key: "2104669f8b71560ddc198939a23191e5"
server_key_iv: "37c2ee9465b73c7bd601cbd755fd3509"
server_key_version: "encryptionkey@0"
I20260927 22:47:35.703228 5109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:35.708392 5109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:35.709095 5109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:35.709326 5109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:35.710146 5109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:35.711372 5109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:35.711475 5109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:35.711576 5109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:35.711644 5109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:35.740228 5109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42541
I20260927 22:47:35.740281 5235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42541 every 8 connection(s)
I20260927 22:47:35.741462 5109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:35.744925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5109
I20260927 22:47:35.745108 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:35.745839 388 external_mini_cluster.cc:1428] Setting key 0b2e4cb5a15b7c27f633a313881bbbcf
I20260927 22:47:35.748225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41579
--builtin_ntp_servers=127.0.97.20:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:47:35.749377 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:35.743914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53399 (local address 127.0.97.62:41579)
0927 22:47:35.744070 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:35.744075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:35.745603 (+ 1528us) server_negotiation.cc:407] Connection header received
0927 22:47:35.747336 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:35.747341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:35.747399 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:35.747517 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:35.748748 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:35.748753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:35.748757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:35.748771 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:35.748798 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:35.748809 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:35.748908 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:35.749073 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 22:47:35.749141 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:47:35.750257 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:35.744446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41579 (local address 127.0.97.1:53399)
0927 22:47:35.745462 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:35.745498 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:35.747093 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:35.747729 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:35.747737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:35.747826 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:35.748474 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:35.748476 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:35.748511 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:35.748598 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:35.748888 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:35.748897 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:35.748904 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:47:35.749142 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 22:47:35.749376 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":315,"threads_started":1}
I20260927 22:47:35.751660 5236 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41579
I20260927 22:47:35.751904 5236 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:35.752415 5236 heartbeater.cc:511] Master 127.0.97.62:41579 requested a full tablet report, sending...
I20260927 22:47:35.754395 5050 ts_manager.cc:194] Registered new tserver with Master: 6f992b0fcdea4981b05faa152af4edcc (127.0.97.1:42541)
W20260927 22:47:36.053916 5239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:36.054198 5239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:36.054309 5239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:36.061558 5239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:36.061666 5239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:36.061807 5239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:36.070684 5239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:36.071990 5239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:36.073731 5239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:36.085791 5244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:36.085868 5247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:36.085819 5245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:36.086706 5239 server_base.cc:1034] running on GCE node
I20260927 22:47:36.087302 5239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:36.088218 5239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:36.089412 5239 hybrid_clock.cc:648] HybridClock initialized: now 1790549256089361 us; error 42 us; skew 500 ppm
I20260927 22:47:36.089742 5239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:36.091943 5239 webserver.cc:460] Webserver started at http://127.0.97.2:42587/ using document root <none> and password file <none>
I20260927 22:47:36.092576 5239 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:36.092695 5239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:36.092994 5239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:36.095870 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "b3a598cb163c45e28811a2b569a90178"
format_stamp: "Formatted at 2026-09-27 22:47:36 on dist-test-slave-kr25"
server_key: "8a45bb34c88e78b58840098b655bf14e"
server_key_iv: "2075a8cc151219008c83c93d06b2bddd"
server_key_version: "encryptionkey@0"
I20260927 22:47:36.096623 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b3a598cb163c45e28811a2b569a90178"
format_stamp: "Formatted at 2026-09-27 22:47:36 on dist-test-slave-kr25"
server_key: "8a45bb34c88e78b58840098b655bf14e"
server_key_iv: "2075a8cc151219008c83c93d06b2bddd"
server_key_version: "encryptionkey@0"
I20260927 22:47:36.101464 5239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:47:36.104797 5253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:36.106113 5239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:47:36.106256 5239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "b3a598cb163c45e28811a2b569a90178"
format_stamp: "Formatted at 2026-09-27 22:47:36 on dist-test-slave-kr25"
server_key: "8a45bb34c88e78b58840098b655bf14e"
server_key_iv: "2075a8cc151219008c83c93d06b2bddd"
server_key_version: "encryptionkey@0"
I20260927 22:47:36.106455 5239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:36.111763 5239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:36.112473 5239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:36.112711 5239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:36.113536 5239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:36.114811 5239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:36.114900 5239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:36.115000 5239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:36.115065 5239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:36.145646 5239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40161
I20260927 22:47:36.145690 5365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40161 every 8 connection(s)
I20260927 22:47:36.146925 5239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:36.154595 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5239
I20260927 22:47:36.154614 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.149338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42509 (local address 127.0.97.62:41579)
0927 22:47:36.149502 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.149506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.150843 (+ 1337us) server_negotiation.cc:407] Connection header received
0927 22:47:36.152525 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.152531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.152595 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.152680 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.153930 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.153936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.153940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.153956 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.153986 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.153998 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.154110 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.154287 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.154349 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:47:36.154754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:36.155202 388 external_mini_cluster.cc:1428] Setting key a06f911ee2a4529fa26a23a14f71db64
I20260927 22:47:36.155615 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.149829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41579 (local address 127.0.97.2:42509)
0927 22:47:36.150683 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.150724 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:36.152252 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:36.152916 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:36.152924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:36.153017 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:36.153634 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:36.153637 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:36.153673 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:36.153769 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:36.154079 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:36.154090 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:36.154098 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:47:36.154342 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 22:47:36.154591 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":238,"threads_started":1}
I20260927 22:47:36.156919 5366 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41579
I20260927 22:47:36.157176 5366 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:36.157670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41579
--builtin_ntp_servers=127.0.97.20:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:47:36.157701 5366 heartbeater.cc:511] Master 127.0.97.62:41579 requested a full tablet report, sending...
I20260927 22:47:36.159003 5050 ts_manager.cc:194] Registered new tserver with Master: b3a598cb163c45e28811a2b569a90178 (127.0.97.2:40161)
W20260927 22:47:36.473418 5369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:36.473692 5369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:36.473779 5369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:36.481215 5369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:36.481321 5369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:36.481437 5369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:36.490744 5369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:36.492028 5369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:36.493815 5369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:36.506400 5374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:36.506433 5377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:36.506466 5375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:36.507146 5369 server_base.cc:1034] running on GCE node
I20260927 22:47:36.507766 5369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:36.508662 5369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:36.509852 5369 hybrid_clock.cc:648] HybridClock initialized: now 1790549256509803 us; error 48 us; skew 500 ppm
I20260927 22:47:36.510152 5369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:36.512319 5369 webserver.cc:460] Webserver started at http://127.0.97.3:38299/ using document root <none> and password file <none>
I20260927 22:47:36.512964 5369 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:36.513060 5369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:36.513357 5369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:36.516141 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "7d96a97fc4b746bfb4412d86472f9a0e"
format_stamp: "Formatted at 2026-09-27 22:47:36 on dist-test-slave-kr25"
server_key: "2de2148155edfa7041239f266e21f230"
server_key_iv: "1e0aa9c6c2c4b0d2f4d46e95c2818f28"
server_key_version: "encryptionkey@0"
I20260927 22:47:36.516911 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7d96a97fc4b746bfb4412d86472f9a0e"
format_stamp: "Formatted at 2026-09-27 22:47:36 on dist-test-slave-kr25"
server_key: "2de2148155edfa7041239f266e21f230"
server_key_iv: "1e0aa9c6c2c4b0d2f4d46e95c2818f28"
server_key_version: "encryptionkey@0"
I20260927 22:47:36.521770 5369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:47:36.525043 5383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:36.526388 5369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:47:36.526522 5369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "7d96a97fc4b746bfb4412d86472f9a0e"
format_stamp: "Formatted at 2026-09-27 22:47:36 on dist-test-slave-kr25"
server_key: "2de2148155edfa7041239f266e21f230"
server_key_iv: "1e0aa9c6c2c4b0d2f4d46e95c2818f28"
server_key_version: "encryptionkey@0"
I20260927 22:47:36.526719 5369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:36.532042 5369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:36.532760 5369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:36.532987 5369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:36.533869 5369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:36.535187 5369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:36.535279 5369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:36.535383 5369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:36.535465 5369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:36.563829 5369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44641
I20260927 22:47:36.563876 5495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44641 every 8 connection(s)
I20260927 22:47:36.565104 5369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:36.572558 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.567529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33959 (local address 127.0.97.62:41579)
0927 22:47:36.567709 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.567713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.568997 (+ 1284us) server_negotiation.cc:407] Connection header received
0927 22:47:36.570634 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.570638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.570690 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.570784 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.571976 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.571980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.571983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.572020 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.572058 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.572072 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.572165 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.572311 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.572359 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:47:36.573473 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.568010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41579 (local address 127.0.97.3:33959)
0927 22:47:36.568867 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.568904 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:36.570385 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:36.570985 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:36.570993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:36.571085 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:36.571706 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:36.571708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:36.571744 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:36.571825 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:36.572149 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:36.572155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:36.572161 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:47:36.572365 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 22:47:36.572596 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":251,"threads_started":1}
I20260927 22:47:36.574699 5496 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41579
I20260927 22:47:36.574954 5496 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:36.575090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5369
I20260927 22:47:36.575254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:36.575595 5496 heartbeater.cc:511] Master 127.0.97.62:41579 requested a full tablet report, sending...
I20260927 22:47:36.575820 388 external_mini_cluster.cc:1428] Setting key 07c83eab7fc7d05a6b09b50c440bd81a
I20260927 22:47:36.577039 5049 ts_manager.cc:194] Registered new tserver with Master: 7d96a97fc4b746bfb4412d86472f9a0e (127.0.97.3:44641)
I20260927 22:47:36.578563 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:47:36.585175 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.582919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.0.97.62:41579)
0927 22:47:36.583065 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.583070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.583566 (+ 496us) server_negotiation.cc:407] Connection header received
0927 22:47:36.583699 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.583702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.583762 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.583840 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.584761 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.584765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.584768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.584782 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.584810 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.584821 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.584908 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.584943 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.584983 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:47:36.588806 5049 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:36.595228 5049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:36.612051 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.605785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44641 (local address 127.0.0.1:57654)
0927 22:47:36.607727 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.607771 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:36.607903 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:36.609009 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:36.609017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:36.609104 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:36.609932 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:36.609935 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:36.609963 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:36.610048 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:36.611292 (+ 1244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:36.611302 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:36.611310 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:47:36.611450 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 22:47:36.611533 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1678,"thread_start_us":2716,"threads_started":1}
I20260927 22:47:36.613060 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.605921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.0.97.3:44641)
0927 22:47:36.608493 (+ 2572us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.608501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.608536 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:47:36.608601 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.608609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.608847 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.609001 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.611052 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.611061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.611076 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.611091 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.611154 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.611173 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.612345 (+ 1172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.612421 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.612514 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2426,"thread_start_us":2260,"threads_started":1}
I20260927 22:47:36.614272 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.608676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.97.1:42541)
0927 22:47:36.611324 (+ 2648us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.611330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.611353 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:47:36.611406 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.611411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.611619 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.612502 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.613544 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.613552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.613564 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.613588 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.613638 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.613655 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.613749 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.613907 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.613991 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2520,"thread_start_us":2402,"threads_started":1}
I20260927 22:47:36.614308 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.605826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40161 (local address 127.0.0.1:52632)
0927 22:47:36.608755 (+ 2929us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.608770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:36.608851 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:36.611533 (+ 2682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:36.611539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:36.611560 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:36.612459 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:36.612462 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:36.612477 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:36.612518 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:36.613130 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:36.613135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:36.613140 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:47:36.613863 (+ 723us) client_negotiation.cc:240] Negotiation successful
0927 22:47:36.613915 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2809,"thread_start_us":1430,"threads_started":1}
I20260927 22:47:36.614578 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.606568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.0.97.2:40161)
0927 22:47:36.611056 (+ 4488us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.611064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.611089 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:47:36.611147 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.611155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.611358 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.611530 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.612897 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.612905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.612918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.612940 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.612988 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.613005 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.614109 (+ 1104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.614179 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.614264 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4342,"thread_start_us":4222,"threads_started":1}
I20260927 22:47:36.614925 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.608180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43250)
0927 22:47:36.610485 (+ 2305us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.610500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:36.610586 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:36.611803 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:36.611808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:36.611828 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:36.613337 (+ 1509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:36.613340 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:36.613356 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:36.613401 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:36.613755 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:36.613759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:36.613764 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:47:36.613910 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 22:47:36.614553 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2195,"thread_start_us":2068,"threads_started":1}
I20260927 22:47:36.616962 5431 tablet_service.cc:1467] Processing CreateTablet for tablet de74a9afb5f044e684fee111b10c65f4 (DEFAULT_TABLE table=test-table [id=15ecd18d2a60499a8fd82fea4eb044bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:36.617624 5301 tablet_service.cc:1467] Processing CreateTablet for tablet de74a9afb5f044e684fee111b10c65f4 (DEFAULT_TABLE table=test-table [id=15ecd18d2a60499a8fd82fea4eb044bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:36.618464 5431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de74a9afb5f044e684fee111b10c65f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:36.618029 5171 tablet_service.cc:1467] Processing CreateTablet for tablet de74a9afb5f044e684fee111b10c65f4 (DEFAULT_TABLE table=test-table [id=15ecd18d2a60499a8fd82fea4eb044bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:36.618706 5301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de74a9afb5f044e684fee111b10c65f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:36.619270 5171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de74a9afb5f044e684fee111b10c65f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:36.628873 5514 tablet_bootstrap.cc:492] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Bootstrap starting.
I20260927 22:47:36.632066 5515 tablet_bootstrap.cc:492] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc: Bootstrap starting.
I20260927 22:47:36.632625 5514 tablet_bootstrap.cc:654] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:36.633447 5516 tablet_bootstrap.cc:492] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Bootstrap starting.
I20260927 22:47:36.633976 5514 log.cc:824] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:36.635089 5515 tablet_bootstrap.cc:654] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:36.636160 5516 tablet_bootstrap.cc:654] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:36.636227 5515 log.cc:824] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:36.636971 5514 tablet_bootstrap.cc:492] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: No bootstrap required, opened a new log
I20260927 22:47:36.637130 5516 log.cc:824] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:36.637223 5514 ts_tablet_manager.cc:1397] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:47:36.639415 5515 tablet_bootstrap.cc:492] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc: No bootstrap required, opened a new log
I20260927 22:47:36.639739 5515 ts_tablet_manager.cc:1397] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 22:47:36.641813 5516 tablet_bootstrap.cc:492] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: No bootstrap required, opened a new log
I20260927 22:47:36.642045 5516 ts_tablet_manager.cc:1397] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:47:36.644197 5514 raft_consensus.cc:348] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.644505 5514 raft_consensus.cc:374] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:36.644590 5514 raft_consensus.cc:729] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a598cb163c45e28811a2b569a90178, State: Initialized, Role: FOLLOWER
I20260927 22:47:36.645119 5514 consensus_queue.cc:260] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.647199 5366 heartbeater.cc:503] Master 127.0.97.62:41579 was elected leader, sending a full tablet report...
I20260927 22:47:36.647792 5514 ts_tablet_manager.cc:1428] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
W20260927 22:47:36.648536 5367 tablet.cc:2369] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:36.649335 5515 raft_consensus.cc:348] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.649717 5515 raft_consensus.cc:374] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:36.649796 5515 raft_consensus.cc:729] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f992b0fcdea4981b05faa152af4edcc, State: Initialized, Role: FOLLOWER
I20260927 22:47:36.650364 5515 consensus_queue.cc:260] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.651916 5236 heartbeater.cc:503] Master 127.0.97.62:41579 was elected leader, sending a full tablet report...
I20260927 22:47:36.651742 5516 raft_consensus.cc:348] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.652114 5516 raft_consensus.cc:374] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:36.652204 5516 raft_consensus.cc:729] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d96a97fc4b746bfb4412d86472f9a0e, State: Initialized, Role: FOLLOWER
I20260927 22:47:36.652477 5515 ts_tablet_manager.cc:1428] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260927 22:47:36.652827 5516 consensus_queue.cc:260] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.653708 5496 heartbeater.cc:503] Master 127.0.97.62:41579 was elected leader, sending a full tablet report...
I20260927 22:47:36.654356 5516 ts_tablet_manager.cc:1428] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 22:47:36.654752 5520 raft_consensus.cc:482] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:36.654984 5520 raft_consensus.cc:504] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.656787 5520 leader_election.cc:288] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d96a97fc4b746bfb4412d86472f9a0e (127.0.97.3:44641), 6f992b0fcdea4981b05faa152af4edcc (127.0.97.1:42541)
I20260927 22:47:36.660432 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.657284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55657 (local address 127.0.97.3:44641)
0927 22:47:36.657584 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.657589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.657824 (+ 235us) server_negotiation.cc:407] Connection header received
0927 22:47:36.658573 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.658578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.658643 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.658713 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.659868 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.659873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.659878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.659896 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.659932 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.659947 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.660045 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.660150 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.660204 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260927 22:47:36.660660 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.656995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44641 (local address 127.0.97.2:55657)
0927 22:47:36.657733 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.657748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:36.658435 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:36.658921 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:36.658926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:36.658947 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:36.659666 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:36.659670 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:36.659688 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:36.659731 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:36.660034 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:36.660039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:36.660044 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:47:36.660140 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:47:36.660189 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":393,"threads_started":1}
I20260927 22:47:36.660708 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.657137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48577 (local address 127.0.97.1:42541)
0927 22:47:36.657286 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.657291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.659063 (+ 1772us) server_negotiation.cc:407] Connection header received
0927 22:47:36.659201 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.659204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.659255 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.659345 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.660161 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.660166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.660169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.660186 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.660218 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.660232 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.660335 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.660424 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.660476 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 22:47:36.660975 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.657303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.97.2:48577)
0927 22:47:36.658972 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.658987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:36.659096 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:36.659368 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:36.659371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:36.659388 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:36.659984 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:47:36.659987 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:36.660000 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:47:36.660034 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:36.660322 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:36.660327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:36.660332 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:47:36.660461 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:47:36.660512 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1560,"thread_start_us":1429,"threads_started":1}
I20260927 22:47:36.661476 5451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de74a9afb5f044e684fee111b10c65f4" candidate_uuid: "b3a598cb163c45e28811a2b569a90178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" is_pre_election: true
I20260927 22:47:36.661816 5191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de74a9afb5f044e684fee111b10c65f4" candidate_uuid: "b3a598cb163c45e28811a2b569a90178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f992b0fcdea4981b05faa152af4edcc" is_pre_election: true
I20260927 22:47:36.661978 5451 raft_consensus.cc:2454] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3a598cb163c45e28811a2b569a90178 in term 0.
I20260927 22:47:36.662273 5191 raft_consensus.cc:2454] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3a598cb163c45e28811a2b569a90178 in term 0.
I20260927 22:47:36.662591 5255 leader_election.cc:302] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d96a97fc4b746bfb4412d86472f9a0e, b3a598cb163c45e28811a2b569a90178; no voters:
I20260927 22:47:36.662914 5520 raft_consensus.cc:2781] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:36.663048 5520 raft_consensus.cc:482] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:36.663149 5520 raft_consensus.cc:3037] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:36.666009 5520 raft_consensus.cc:504] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.666610 5520 leader_election.cc:288] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [CANDIDATE]: Term 1 election: Requested vote from peers 7d96a97fc4b746bfb4412d86472f9a0e (127.0.97.3:44641), 6f992b0fcdea4981b05faa152af4edcc (127.0.97.1:42541)
I20260927 22:47:36.666944 5451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de74a9afb5f044e684fee111b10c65f4" candidate_uuid: "b3a598cb163c45e28811a2b569a90178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d96a97fc4b746bfb4412d86472f9a0e"
I20260927 22:47:36.667125 5451 raft_consensus.cc:3037] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:36.668057 5191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de74a9afb5f044e684fee111b10c65f4" candidate_uuid: "b3a598cb163c45e28811a2b569a90178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f992b0fcdea4981b05faa152af4edcc"
I20260927 22:47:36.668242 5191 raft_consensus.cc:3037] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:36.670105 5451 raft_consensus.cc:2454] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3a598cb163c45e28811a2b569a90178 in term 1.
I20260927 22:47:36.670436 5255 leader_election.cc:302] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d96a97fc4b746bfb4412d86472f9a0e, b3a598cb163c45e28811a2b569a90178; no voters:
I20260927 22:47:36.670706 5520 raft_consensus.cc:2781] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:36.671177 5520 raft_consensus.cc:686] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 1 LEADER]: Becoming Leader. State: Replica: b3a598cb163c45e28811a2b569a90178, State: Running, Role: LEADER
I20260927 22:47:36.671229 5191 raft_consensus.cc:2454] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3a598cb163c45e28811a2b569a90178 in term 1.
I20260927 22:47:36.671706 5520 consensus_queue.cc:237] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } }
I20260927 22:47:36.676007 5050 catalog_manager.cc:5495] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3a598cb163c45e28811a2b569a90178 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b3a598cb163c45e28811a2b569a90178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3a598cb163c45e28811a2b569a90178" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:36.694428 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.692323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.0.97.2:40161)
0927 22:47:36.692487 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.692492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.692665 (+ 173us) server_negotiation.cc:407] Connection header received
0927 22:47:36.692835 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.692839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.692912 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.692997 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.693877 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.693884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.693888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.693906 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.693934 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.693946 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.694047 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.694139 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.694197 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:47:36.702142 5191 raft_consensus.cc:1261] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc [term 1 FOLLOWER]: Refusing update from remote peer b3a598cb163c45e28811a2b569a90178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:36.702152 5451 raft_consensus.cc:1261] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 1 FOLLOWER]: Refusing update from remote peer b3a598cb163c45e28811a2b569a90178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:36.703015 5520 consensus_queue.cc:1035] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f992b0fcdea4981b05faa152af4edcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:36.703265 5525 consensus_queue.cc:1035] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d96a97fc4b746bfb4412d86472f9a0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:36.715663 5529 mvcc.cc:205] Tried to move back new op lower bound from 7334089755442401280 to 7334089755330863104. Current Snapshot: MvccSnapshot[applied={T|T < 7334089755442401280}]
I20260927 22:47:36.731500 5050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:36.731739 5050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:36.734728 5050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbdcdae86596410eba784419496604b1: Sending DeleteTablet for 3 replicas of tablet de74a9afb5f044e684fee111b10c65f4
I20260927 22:47:36.735838 5431 tablet_service.cc:1514] Processing DeleteTablet for tablet de74a9afb5f044e684fee111b10c65f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:36 UTC) from {username='slave'} at 127.0.0.1:57654
I20260927 22:47:36.735894 5171 tablet_service.cc:1514] Processing DeleteTablet for tablet de74a9afb5f044e684fee111b10c65f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:36 UTC) from {username='slave'} at 127.0.0.1:43250
I20260927 22:47:36.736687 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet de74a9afb5f044e684fee111b10c65f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:36 UTC) from {username='slave'} at 127.0.0.1:52632
I20260927 22:47:36.736958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5109
I20260927 22:47:36.750622 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.738222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.0.97.1:42541)
0927 22:47:36.738383 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.738389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.748066 (+ 9677us) server_negotiation.cc:407] Connection header received
0927 22:47:36.748476 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.748484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.748589 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.748657 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.749835 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.749842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.749847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.749867 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.749908 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.749923 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.750016 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.750229 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.750291 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 22:47:36.751613 5231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:36.751700 5540 tablet_replica.cc:330] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: stopping tablet replica
I20260927 22:47:36.752146 5540 raft_consensus.cc:2229] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:36.752889 5540 raft_consensus.cc:2258] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:36.753770 5538 tablet_replica.cc:330] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: stopping tablet replica
I20260927 22:47:36.754182 5538 raft_consensus.cc:2229] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:36.754666 5538 raft_consensus.cc:2258] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:36.755369 5540 ts_tablet_manager.cc:1905] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:36.756717 5538 ts_tablet_manager.cc:1905] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:36.763345 5538 ts_tablet_manager.cc:1918] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:36.763971 5540 ts_tablet_manager.cc:1918] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:36.764324 5540 log.cc:1196] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/de74a9afb5f044e684fee111b10c65f4
I20260927 22:47:36.764943 5540 ts_tablet_manager.cc:1939] T de74a9afb5f044e684fee111b10c65f4 P b3a598cb163c45e28811a2b569a90178: Deleting consensus metadata
I20260927 22:47:36.765640 5538 log.cc:1196] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/de74a9afb5f044e684fee111b10c65f4
I20260927 22:47:36.766592 5538 ts_tablet_manager.cc:1939] T de74a9afb5f044e684fee111b10c65f4 P 7d96a97fc4b746bfb4412d86472f9a0e: Deleting consensus metadata
I20260927 22:47:36.767364 5037 catalog_manager.cc:4841] TS b3a598cb163c45e28811a2b569a90178 (127.0.97.2:40161): tablet de74a9afb5f044e684fee111b10c65f4 (table test-table [id=15ecd18d2a60499a8fd82fea4eb044bb]) successfully deleted
I20260927 22:47:36.768234 5035 catalog_manager.cc:4841] TS 7d96a97fc4b746bfb4412d86472f9a0e (127.0.97.3:44641): tablet de74a9afb5f044e684fee111b10c65f4 (table test-table [id=15ecd18d2a60499a8fd82fea4eb044bb]) successfully deleted
I20260927 22:47:36.850978 5539 tablet_replica.cc:330] T de74a9afb5f044e684fee111b10c65f4 P 6f992b0fcdea4981b05faa152af4edcc: stopping tablet replica
I20260927 22:47:36.850967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5109
I20260927 22:47:36.864508 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5239
W20260927 22:47:36.864537 5037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42541 (error 108)
I20260927 22:47:36.866879 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.866405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43268)
0927 22:47:36.866590 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.866722 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 22:47:36.867184 5037 catalog_manager.cc:4584] TS 6f992b0fcdea4981b05faa152af4edcc (127.0.97.1:42541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de74a9afb5f044e684fee111b10c65f4: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
I20260927 22:47:36.867815 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.865294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.97.2:40161)
0927 22:47:36.865470 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.865476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.865799 (+ 323us) server_negotiation.cc:407] Connection header received
0927 22:47:36.865892 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.865898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.865960 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.866048 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.867091 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.867096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.867100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.867121 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.867159 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.867179 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.867308 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.867480 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.867533 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:47:36.868448 5361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:36.920822 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.920294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43276)
0927 22:47:36.920513 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.920642 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:47:36.921785 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.921419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43282)
0927 22:47:36.921560 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.921648 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 22:47:36.972129 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5239
I20260927 22:47:36.982142 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.981680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43298)
0927 22:47:36.981877 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.981976 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:47:36.983109 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.982749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43306)
0927 22:47:36.982894 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:36.982999 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 22:47:36.986332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5369
I20260927 22:47:36.988665 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:36.986955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.97.3:44641)
0927 22:47:36.987108 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:36.987113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:36.987145 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:47:36.987276 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:36.987280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:36.987362 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:36.987459 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:36.988154 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:36.988158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:36.988161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:36.988174 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:36.988204 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:36.988215 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:36.988310 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:36.988360 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 22:47:36.988400 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:47:36.989203 5491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:37.063511 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:37.062889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43314)
0927 22:47:37.063158 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:37.063277 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:47:37.064641 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:37.064167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42541 (local address 127.0.0.1:43322)
0927 22:47:37.064412 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:37.064492 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 22:47:37.074625 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5369
I20260927 22:47:37.087085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5019
I20260927 22:47:37.088239 5080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:37.190562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5019
2026-09-27T22:47:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2386 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 22:47:37.226297 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:47:37.228729 388 webserver.cc:460] Webserver started at https://127.0.0.1:35409/ using document root <none> and password file <none>
I20260927 22:47:37.228976 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:47:37.229501 388 webserver.cc:460] Webserver started at http://127.0.0.1:40197/ using document root <none> and password file <none>
2026-09-27T22:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:37Z Disabled control of system clock
I20260927 22:47:37.251596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44113
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44113
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35409/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/testchainca.cert with env {}
W20260927 22:47:37.559580 5552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:37.559847 5552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:37.559932 5552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:37.566788 5552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:37.566881 5552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:37.566917 5552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:37.566972 5552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:37.567013 5552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:47:37.567076 5552 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:47:37.567116 5552 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35409/jwks.json
I20260927 22:47:37.576571 5552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35409/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:37.577759 5552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:37.578929 5552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:37.590086 5558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:37.590076 5557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:37.590130 5560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:37.590819 5552 server_base.cc:1034] running on GCE node
I20260927 22:47:37.591634 5552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:37.592929 5552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:37.594130 5552 hybrid_clock.cc:648] HybridClock initialized: now 1790549257594085 us; error 37 us; skew 500 ppm
I20260927 22:47:37.594444 5552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:37.596556 5552 webserver.cc:460] Webserver started at http://127.0.97.62:36235/ using document root <none> and password file <none>
I20260927 22:47:37.597182 5552 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:37.597277 5552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:37.597545 5552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:37.600252 5552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "50e42cf0578742bd8e71c9c2fa29720e"
format_stamp: "Formatted at 2026-09-27 22:47:37 on dist-test-slave-kr25"
server_key: "a535048fa3208ef34bdd08f2ed0f772c"
server_key_iv: "74cd8c731a7ea21bdec2b3b5ac369194"
server_key_version: "encryptionkey@0"
I20260927 22:47:37.600963 5552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "50e42cf0578742bd8e71c9c2fa29720e"
format_stamp: "Formatted at 2026-09-27 22:47:37 on dist-test-slave-kr25"
server_key: "a535048fa3208ef34bdd08f2ed0f772c"
server_key_iv: "74cd8c731a7ea21bdec2b3b5ac369194"
server_key_version: "encryptionkey@0"
I20260927 22:47:37.605535 5552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:37.608685 5566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:37.609952 5552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:47:37.610105 5552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "50e42cf0578742bd8e71c9c2fa29720e"
format_stamp: "Formatted at 2026-09-27 22:47:37 on dist-test-slave-kr25"
server_key: "a535048fa3208ef34bdd08f2ed0f772c"
server_key_iv: "74cd8c731a7ea21bdec2b3b5ac369194"
server_key_version: "encryptionkey@0"
I20260927 22:47:37.610284 5552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:37.637555 5552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:37.638350 5552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:37.638600 5552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:37.654583 5552 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44113
I20260927 22:47:37.654603 5617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44113 every 8 connection(s)
I20260927 22:47:37.655887 5552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:37.657016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5552
I20260927 22:47:37.657207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:37.657928 388 external_mini_cluster.cc:1428] Setting key 8f1f2ea5890aa4d961f722d8c7255d06
I20260927 22:47:37.660382 5618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:37.670231 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:37.659737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.0.97.62:44113)
0927 22:47:37.660605 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:37.660629 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:37.660672 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:47:37.662249 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:37.662269 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:37.662736 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:37.663076 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:37.663637 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:37.664969 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:37.666237 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:37.666845 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:37.666997 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:37.668163 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:37.668170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:37.668183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:37.668718 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:37.668788 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:37.668810 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:37.668904 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:37.669118 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 22:47:37.669373 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260927 22:47:37.670620 5618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e: Bootstrap starting.
I20260927 22:47:37.673657 5618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:37.673676 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:47:37.674551 5618 log.cc:824] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:37.677150 5618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e: No bootstrap required, opened a new log
I20260927 22:47:37.683809 5618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e42cf0578742bd8e71c9c2fa29720e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260927 22:47:37.684116 5618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:37.684211 5618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e42cf0578742bd8e71c9c2fa29720e, State: Initialized, Role: FOLLOWER
I20260927 22:47:37.684738 5618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e42cf0578742bd8e71c9c2fa29720e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260927 22:47:37.684935 5618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:37.685052 5618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:37.685199 5618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:37.687079 5618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e42cf0578742bd8e71c9c2fa29720e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260927 22:47:37.687548 5618 leader_election.cc:302] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50e42cf0578742bd8e71c9c2fa29720e; no voters:
I20260927 22:47:37.688046 5618 leader_election.cc:288] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:37.688210 5623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:37.689198 5623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [term 1 LEADER]: Becoming Leader. State: Replica: 50e42cf0578742bd8e71c9c2fa29720e, State: Running, Role: LEADER
I20260927 22:47:37.689677 5623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e42cf0578742bd8e71c9c2fa29720e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260927 22:47:37.689982 5618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:37.692251 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50e42cf0578742bd8e71c9c2fa29720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e42cf0578742bd8e71c9c2fa29720e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } } }
I20260927 22:47:37.692194 5625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50e42cf0578742bd8e71c9c2fa29720e. Latest consensus state: current_term: 1 leader_uuid: "50e42cf0578742bd8e71c9c2fa29720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e42cf0578742bd8e71c9c2fa29720e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } } }
I20260927 22:47:37.692549 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:37.692549 5625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:37.693300 5630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:37.697808 5630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:37.706815 5630 catalog_manager.cc:1347] Generated new cluster ID: cc45fd2b32984569a7058294c470e33b
I20260927 22:47:37.706921 5630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:37.726672 5630 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:37.727908 5630 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:37.735497 5630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50e42cf0578742bd8e71c9c2fa29720e: Generated new TSK 0
I20260927 22:47:37.736338 5630 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:47:37.774322 5648 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 22:47:37.774475 5648 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:47:37.775696 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:37.754360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.0.97.62:44113)
0927 22:47:37.754526 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:37.754532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:37.755024 (+ 492us) server_negotiation.cc:407] Connection header received
0927 22:47:37.755307 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:37.755313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:37.755395 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:37.755564 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:47:37.756029 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:37.757128 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:37.758384 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:37.759074 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:37.759191 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:37.759642 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:47:37.774949 (+ 15307us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35409/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:47:37.775314 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35409/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:47:37.775818 5621 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35409/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 22:47:37.776993 5647 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:37.754352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44113 (local address 127.0.0.1:56758)
0927 22:47:37.754938 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:37.754953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:37.755094 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:37.755588 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:37.755591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:37.755601 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:47:37.755896 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:37.755903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:37.757230 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:37.757233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:37.758261 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:37.758272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:37.759171 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:37.759205 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:37.759330 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:37.759378 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:47:37.775234 (+ 15856us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35409/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:47:37.775560 (+ 326us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35409/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":292,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:44113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35409/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 22:47:37.778921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5552
I20260927 22:47:37.780027 5613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:37.870460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5552
2026-09-27T22:47:37Z chronyd exiting
I20260927 22:47:37.916893 388 test_util.cc:175] -----------------------------------------------
I20260927 22:47:37.917032 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549216186161-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (694 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 22:47:37.920562 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:47:37.922315 388 webserver.cc:460] Webserver started at https://127.0.0.1:39489/ using document root <none> and password file <none>
I20260927 22:47:37.922547 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:47:37.923138 388 webserver.cc:460] Webserver started at http://127.0.0.1:46239/ using document root <none> and password file <none>
2026-09-27T22:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:37Z Disabled control of system clock
I20260927 22:47:37.946022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39865
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39489/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/testchainca.cert with env {}
W20260927 22:47:38.247128 5655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:38.247394 5655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:38.247498 5655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:38.254787 5655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:38.254907 5655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:38.254952 5655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:38.255007 5655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:38.255051 5655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:47:38.255115 5655 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:47:38.255152 5655 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39489/jwks.json
I20260927 22:47:38.265084 5655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39489/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:38.266281 5655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:38.267505 5655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:38.277951 5660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:38.278319 5661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:38.278532 5663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:38.279174 5655 server_base.cc:1034] running on GCE node
I20260927 22:47:38.279888 5655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:38.281109 5655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:38.282300 5655 hybrid_clock.cc:648] HybridClock initialized: now 1790549258282263 us; error 34 us; skew 500 ppm
I20260927 22:47:38.282584 5655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:38.284596 5655 webserver.cc:460] Webserver started at http://127.0.97.62:43467/ using document root <none> and password file <none>
I20260927 22:47:38.285218 5655 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:38.285321 5655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:38.285586 5655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:38.288311 5655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "e593a26ed85f4999a85f46caf96c3b2e"
format_stamp: "Formatted at 2026-09-27 22:47:38 on dist-test-slave-kr25"
server_key: "a10cdfdd48d429b21e0d5c2b7e653705"
server_key_iv: "8e950346cabd1146ca5ba7a3de3d3348"
server_key_version: "encryptionkey@0"
I20260927 22:47:38.289011 5655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "e593a26ed85f4999a85f46caf96c3b2e"
format_stamp: "Formatted at 2026-09-27 22:47:38 on dist-test-slave-kr25"
server_key: "a10cdfdd48d429b21e0d5c2b7e653705"
server_key_iv: "8e950346cabd1146ca5ba7a3de3d3348"
server_key_version: "encryptionkey@0"
I20260927 22:47:38.293637 5655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:47:38.296798 5669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:38.298031 5655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:47:38.298206 5655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "e593a26ed85f4999a85f46caf96c3b2e"
format_stamp: "Formatted at 2026-09-27 22:47:38 on dist-test-slave-kr25"
server_key: "a10cdfdd48d429b21e0d5c2b7e653705"
server_key_iv: "8e950346cabd1146ca5ba7a3de3d3348"
server_key_version: "encryptionkey@0"
I20260927 22:47:38.298381 5655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:38.311774 5655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:38.312510 5655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:38.312754 5655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:38.328622 5655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39865
I20260927 22:47:38.328590 5720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39865 every 8 connection(s)
I20260927 22:47:38.329890 5655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:38.330889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5655
I20260927 22:47:38.331067 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:38.331693 388 external_mini_cluster.cc:1428] Setting key 8b26f5f762fe039834277601544f1d2f
I20260927 22:47:38.333999 5721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:38.346649 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:38.333493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.0.97.62:39865)
0927 22:47:38.334486 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:38.334500 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:38.334549 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:47:38.336657 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:38.336685 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:38.337300 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:38.337724 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:38.338242 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:38.340231 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:38.341522 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:38.342486 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:38.342663 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:38.343868 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:38.343878 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:38.343895 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:38.344674 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:38.344764 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:38.344789 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:38.344908 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:38.345108 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 22:47:38.345454 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":283,"threads_started":1}
I20260927 22:47:38.347456 5721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e: Bootstrap starting.
I20260927 22:47:38.350440 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:47:38.350580 5721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:38.351711 5721 log.cc:824] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:38.354308 5721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e: No bootstrap required, opened a new log
I20260927 22:47:38.360929 5721 raft_consensus.cc:348] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e593a26ed85f4999a85f46caf96c3b2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39865 } }
I20260927 22:47:38.361232 5721 raft_consensus.cc:374] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:38.361303 5721 raft_consensus.cc:729] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e593a26ed85f4999a85f46caf96c3b2e, State: Initialized, Role: FOLLOWER
I20260927 22:47:38.361797 5721 consensus_queue.cc:260] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e593a26ed85f4999a85f46caf96c3b2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39865 } }
I20260927 22:47:38.361987 5721 raft_consensus.cc:388] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:38.362078 5721 raft_consensus.cc:482] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:38.362201 5721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:38.364192 5721 raft_consensus.cc:504] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e593a26ed85f4999a85f46caf96c3b2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39865 } }
I20260927 22:47:38.364611 5721 leader_election.cc:302] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e593a26ed85f4999a85f46caf96c3b2e; no voters:
I20260927 22:47:38.365103 5721 leader_election.cc:288] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:38.365242 5726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:38.366523 5726 raft_consensus.cc:686] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [term 1 LEADER]: Becoming Leader. State: Replica: e593a26ed85f4999a85f46caf96c3b2e, State: Running, Role: LEADER
I20260927 22:47:38.367072 5726 consensus_queue.cc:237] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e593a26ed85f4999a85f46caf96c3b2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39865 } }
I20260927 22:47:38.367383 5721 sys_catalog.cc:564] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:38.369997 5728 sys_catalog.cc:455] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e593a26ed85f4999a85f46caf96c3b2e. Latest consensus state: current_term: 1 leader_uuid: "e593a26ed85f4999a85f46caf96c3b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e593a26ed85f4999a85f46caf96c3b2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39865 } } }
I20260927 22:47:38.370420 5728 sys_catalog.cc:458] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:38.370321 5727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e593a26ed85f4999a85f46caf96c3b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e593a26ed85f4999a85f46caf96c3b2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39865 } } }
I20260927 22:47:38.370620 5727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:38.371317 5735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:38.377497 5735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:38.388979 5735 catalog_manager.cc:1347] Generated new cluster ID: 191cf1f598554a74a5e95380be5d8f0b
I20260927 22:47:38.389098 5735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:38.405700 5735 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:38.406929 5735 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:38.418187 5735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e593a26ed85f4999a85f46caf96c3b2e: Generated new TSK 0
I20260927 22:47:38.419037 5735 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:47:38.458528 5751 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 22:47:38.458638 5751 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 22:47:38.459924 5750 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:38.440302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39865 (local address 127.0.0.1:45944)
0927 22:47:38.440839 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:38.440855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:38.440970 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:38.441395 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:38.441398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:38.441407 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:47:38.441656 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:38.441663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:38.442990 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:38.442993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:38.444086 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:38.444099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:38.445094 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:38.445146 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:38.445282 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:38.445312 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:47:38.459454 (+ 14142us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:47:38.459646 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":293,"threads_started":1}
I20260927 22:47:38.459939 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:38.440464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.0.97.62:39865)
0927 22:47:38.440652 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:38.440658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:38.440926 (+ 268us) server_negotiation.cc:407] Connection header received
0927 22:47:38.441145 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:38.441152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:38.441237 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:38.441395 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:47:38.441781 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:38.442882 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:38.444277 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:38.444988 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:38.445117 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:38.445584 (+ 467us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:47:38.459225 (+ 13641us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:47:38.459634 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:47:38.460052 5724 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 22:47:38.460587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5655
I20260927 22:47:38.461668 5716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:38.552881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5655
2026-09-27T22:47:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (683 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 22:47:38.604507 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:47:38.606245 388 webserver.cc:460] Webserver started at https://127.0.0.1:44901/ using document root <none> and password file <none>
I20260927 22:47:38.606431 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:47:38.606979 388 webserver.cc:460] Webserver started at http://127.0.0.1:39821/ using document root <none> and password file <none>
2026-09-27T22:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:38Z Disabled control of system clock
I20260927 22:47:38.630046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46277
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46277
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44901/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:47:38.930505 5758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:38.930774 5758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:38.930866 5758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:38.937956 5758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:38.938077 5758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:38.938128 5758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:38.938184 5758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:38.938225 5758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:47:38.938283 5758 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:47:38.938320 5758 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44901/jwks.json
I20260927 22:47:38.947274 5758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44901/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:38.948477 5758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:38.950138 5758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:38.960553 5766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:38.960620 5763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:38.960596 5764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:38.961333 5758 server_base.cc:1034] running on GCE node
I20260927 22:47:38.961995 5758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:38.963198 5758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:38.964385 5758 hybrid_clock.cc:648] HybridClock initialized: now 1790549258964322 us; error 54 us; skew 500 ppm
I20260927 22:47:38.964718 5758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:38.966703 5758 webserver.cc:460] Webserver started at http://127.0.97.62:46073/ using document root <none> and password file <none>
I20260927 22:47:38.967317 5758 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:38.967410 5758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:38.967727 5758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:38.970378 5758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "b23b10c608a04cceb95f2716eac2ac63"
format_stamp: "Formatted at 2026-09-27 22:47:38 on dist-test-slave-kr25"
server_key: "638bb57ff49e1fcc90676e1316d183a1"
server_key_iv: "1d3e78163ebf5dbd02857004f6a56222"
server_key_version: "encryptionkey@0"
I20260927 22:47:38.971084 5758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "b23b10c608a04cceb95f2716eac2ac63"
format_stamp: "Formatted at 2026-09-27 22:47:38 on dist-test-slave-kr25"
server_key: "638bb57ff49e1fcc90676e1316d183a1"
server_key_iv: "1d3e78163ebf5dbd02857004f6a56222"
server_key_version: "encryptionkey@0"
I20260927 22:47:38.975818 5758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:47:38.978988 5772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:38.980264 5758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:38.980417 5758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "b23b10c608a04cceb95f2716eac2ac63"
format_stamp: "Formatted at 2026-09-27 22:47:38 on dist-test-slave-kr25"
server_key: "638bb57ff49e1fcc90676e1316d183a1"
server_key_iv: "1d3e78163ebf5dbd02857004f6a56222"
server_key_version: "encryptionkey@0"
I20260927 22:47:38.980599 5758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:38.994855 5758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:38.995622 5758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:38.995850 5758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:39.011595 5758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46277
I20260927 22:47:39.011615 5823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46277 every 8 connection(s)
I20260927 22:47:39.012884 5758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:39.015087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5758
I20260927 22:47:39.015277 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:39.015915 388 external_mini_cluster.cc:1428] Setting key 49a19f55deb435e6ba4d44393cfba98b
I20260927 22:47:39.016737 5824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:39.030959 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:39.017738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.97.62:46277)
0927 22:47:39.018757 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:39.018783 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:39.018832 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:47:39.021199 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:39.021234 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:39.021871 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:39.022314 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:39.022882 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.024844 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.026107 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.027099 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.027276 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:39.028275 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:39.028285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:39.028304 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:39.029024 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:39.029121 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:39.029146 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:39.029245 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:39.029410 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 22:47:39.029722 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":330,"threads_started":1}
I20260927 22:47:39.031203 5824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63: Bootstrap starting.
I20260927 22:47:39.034404 5824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:39.034498 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:47:39.035349 5824 log.cc:824] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:39.038130 5824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63: No bootstrap required, opened a new log
I20260927 22:47:39.045178 5824 raft_consensus.cc:348] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23b10c608a04cceb95f2716eac2ac63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46277 } }
I20260927 22:47:39.045506 5824 raft_consensus.cc:374] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:39.045598 5824 raft_consensus.cc:729] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23b10c608a04cceb95f2716eac2ac63, State: Initialized, Role: FOLLOWER
I20260927 22:47:39.046100 5824 consensus_queue.cc:260] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23b10c608a04cceb95f2716eac2ac63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46277 } }
I20260927 22:47:39.046308 5824 raft_consensus.cc:388] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:39.046424 5824 raft_consensus.cc:482] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:39.046561 5824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:39.048662 5824 raft_consensus.cc:504] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23b10c608a04cceb95f2716eac2ac63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46277 } }
I20260927 22:47:39.049136 5824 leader_election.cc:302] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b23b10c608a04cceb95f2716eac2ac63; no voters:
I20260927 22:47:39.049741 5824 leader_election.cc:288] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:39.049887 5829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:39.050558 5829 raft_consensus.cc:686] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [term 1 LEADER]: Becoming Leader. State: Replica: b23b10c608a04cceb95f2716eac2ac63, State: Running, Role: LEADER
I20260927 22:47:39.051062 5829 consensus_queue.cc:237] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23b10c608a04cceb95f2716eac2ac63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46277 } }
I20260927 22:47:39.051723 5824 sys_catalog.cc:564] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:39.053524 5831 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b23b10c608a04cceb95f2716eac2ac63. Latest consensus state: current_term: 1 leader_uuid: "b23b10c608a04cceb95f2716eac2ac63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23b10c608a04cceb95f2716eac2ac63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46277 } } }
I20260927 22:47:39.053707 5830 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b23b10c608a04cceb95f2716eac2ac63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23b10c608a04cceb95f2716eac2ac63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46277 } } }
I20260927 22:47:39.053891 5831 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:39.053954 5830 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:39.060479 5836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:39.065104 5836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:39.073112 5836 catalog_manager.cc:1347] Generated new cluster ID: 64d21acbcbce4ddda28ea55a49560751
I20260927 22:47:39.073225 5836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:39.095178 5836 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:39.097008 5836 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:39.110389 5836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b23b10c608a04cceb95f2716eac2ac63: Generated new TSK 0
I20260927 22:47:39.111416 5836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:47:39.132186 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:39.124369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.0.97.62:46277)
0927 22:47:39.124505 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:39.124510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:39.124835 (+ 325us) server_negotiation.cc:407] Connection header received
0927 22:47:39.125022 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:39.125028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:39.125115 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:39.125282 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:47:39.125665 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.126809 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.128122 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.128798 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.128920 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:39.129363 (+ 443us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:47:39.131589 (+ 2226us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:47:39.131893 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 22:47:39.132309 5827 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 22:47:39.133816 5853 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:39.124213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46277 (local address 127.0.0.1:48944)
0927 22:47:39.124742 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:39.124757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:39.124870 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:39.125277 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:39.125280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:39.125288 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:47:39.125535 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:39.125542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.126920 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.126924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:39.127998 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:39.128010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.128895 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.128952 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:39.129074 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:39.129105 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:47:39.131769 (+ 2664us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:47:39.133393 (+ 1624us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46277: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":1351,"thread_start_us":285,"threads_started":1}
I20260927 22:47:39.135102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5758
I20260927 22:47:39.136082 5819 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 22:47:39.143253 5854 webserver.cc:533] Webserver: sslize failed:
E20260927 22:47:39.143345 5854 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:47:39.225224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5758
2026-09-27T22:47:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (671 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T22:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:39Z Disabled control of system clock
I20260927 22:47:39.297631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36655
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36655
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/testchainca.cert with env {}
W20260927 22:47:39.604163 5859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:39.604434 5859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:39.604564 5859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:39.611416 5859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:39.611559 5859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:39.611625 5859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:39.611699 5859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:39.611757 5859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:47:39.611837 5859 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:47:39.611888 5859 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 22:47:39.621470 5859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:39.622742 5859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:39.624053 5859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:39.634742 5865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:39.634811 5864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:39.634788 5867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:39.635391 5859 server_base.cc:1034] running on GCE node
I20260927 22:47:39.636098 5859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:39.637322 5859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:39.638501 5859 hybrid_clock.cc:648] HybridClock initialized: now 1790549259638462 us; error 34 us; skew 500 ppm
I20260927 22:47:39.638801 5859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:47:39.644780 5859 webserver.cc:460] Webserver started at http://127.0.97.62:38491/ using document root <none> and password file <none>
I20260927 22:47:39.645445 5859 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:39.645551 5859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:39.645859 5859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:39.648703 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "f006f1f7f256495f870ab3ea47f4d590"
format_stamp: "Formatted at 2026-09-27 22:47:39 on dist-test-slave-kr25"
server_key: "2796b18dafb6a36838c99f065961a373"
server_key_iv: "d9432784c18683a7cf3f66a0eb65cda1"
server_key_version: "encryptionkey@0"
I20260927 22:47:39.649519 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "f006f1f7f256495f870ab3ea47f4d590"
format_stamp: "Formatted at 2026-09-27 22:47:39 on dist-test-slave-kr25"
server_key: "2796b18dafb6a36838c99f065961a373"
server_key_iv: "d9432784c18683a7cf3f66a0eb65cda1"
server_key_version: "encryptionkey@0"
I20260927 22:47:39.654516 5859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:47:39.657893 5873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:39.659189 5859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:39.659351 5859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "f006f1f7f256495f870ab3ea47f4d590"
format_stamp: "Formatted at 2026-09-27 22:47:39 on dist-test-slave-kr25"
server_key: "2796b18dafb6a36838c99f065961a373"
server_key_iv: "d9432784c18683a7cf3f66a0eb65cda1"
server_key_version: "encryptionkey@0"
I20260927 22:47:39.659590 5859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:39.676265 5859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:39.677062 5859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:39.677322 5859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:39.693789 5859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36655
I20260927 22:47:39.693806 5924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36655 every 8 connection(s)
I20260927 22:47:39.695031 5859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:39.698521 5925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:39.704387 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5859
I20260927 22:47:39.704547 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:39.705137 388 external_mini_cluster.cc:1428] Setting key 0dbc9ba7859c894212e3b52c734b8959
I20260927 22:47:39.708638 5925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590: Bootstrap starting.
I20260927 22:47:39.711778 5925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:39.712841 5925 log.cc:824] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:39.715354 5925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590: No bootstrap required, opened a new log
I20260927 22:47:39.717953 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:39.706994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.0.97.62:36655)
0927 22:47:39.707949 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:39.707960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:39.707998 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:47:39.709621 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:39.709644 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:39.710102 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:39.710412 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:39.710953 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.712272 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.713882 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.714528 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.714720 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:39.715795 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:39.715803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:39.715816 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:47:39.716311 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:39.716383 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:47:39.716402 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:39.716498 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:39.716727 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 22:47:39.717006 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":365,"threads_started":1}
I20260927 22:47:39.721735 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:47:39.723343 5925 raft_consensus.cc:348] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006f1f7f256495f870ab3ea47f4d590" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36655 } }
I20260927 22:47:39.723719 5925 raft_consensus.cc:374] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:39.723837 5925 raft_consensus.cc:729] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006f1f7f256495f870ab3ea47f4d590, State: Initialized, Role: FOLLOWER
I20260927 22:47:39.724393 5925 consensus_queue.cc:260] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006f1f7f256495f870ab3ea47f4d590" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36655 } }
I20260927 22:47:39.724637 5925 raft_consensus.cc:388] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:39.724754 5925 raft_consensus.cc:482] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:39.724901 5925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:39.726866 5925 raft_consensus.cc:504] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006f1f7f256495f870ab3ea47f4d590" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36655 } }
I20260927 22:47:39.727346 5925 leader_election.cc:302] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f006f1f7f256495f870ab3ea47f4d590; no voters:
I20260927 22:47:39.727960 5925 leader_election.cc:288] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:39.728077 5930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:39.728708 5930 raft_consensus.cc:686] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [term 1 LEADER]: Becoming Leader. State: Replica: f006f1f7f256495f870ab3ea47f4d590, State: Running, Role: LEADER
I20260927 22:47:39.729187 5930 consensus_queue.cc:237] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006f1f7f256495f870ab3ea47f4d590" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36655 } }
I20260927 22:47:39.729704 5925 sys_catalog.cc:564] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:39.731722 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f006f1f7f256495f870ab3ea47f4d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006f1f7f256495f870ab3ea47f4d590" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36655 } } }
I20260927 22:47:39.732111 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:39.732111 5932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f006f1f7f256495f870ab3ea47f4d590. Latest consensus state: current_term: 1 leader_uuid: "f006f1f7f256495f870ab3ea47f4d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006f1f7f256495f870ab3ea47f4d590" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36655 } } }
I20260927 22:47:39.732409 5932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:39.734714 5939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:39.739498 5939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:39.747731 5939 catalog_manager.cc:1347] Generated new cluster ID: 4edfa76303e148e68e94d0e6804cac23
I20260927 22:47:39.747853 5939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:39.764933 5939 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:39.766096 5939 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:39.771742 5939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f006f1f7f256495f870ab3ea47f4d590: Generated new TSK 0
I20260927 22:47:39.772544 5939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:47:39.858537 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:39.793052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.97.62:36655)
0927 22:47:39.793240 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:39.793246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:39.793691 (+ 445us) server_negotiation.cc:407] Connection header received
0927 22:47:39.793894 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:39.793900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:39.793985 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:39.794144 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:47:39.794518 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.795669 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.796960 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.797614 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.797735 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:39.798160 (+ 425us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:47:39.857819 (+ 59659us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:47:39.858217 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34436: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:47:39.858677 5928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34436: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 22:47:39.858693 5954 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:39.792988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36655 (local address 127.0.0.1:34436)
0927 22:47:39.793593 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:39.793609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:39.793723 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:39.794155 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:39.794158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:39.794167 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:47:39.794389 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:39.794396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.795796 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.795800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:39.796826 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:39.796838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:39.797727 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:39.797773 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:39.797886 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:39.797920 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:47:39.858138 (+ 60218us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:47:39.858383 (+ 245us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36655: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":323,"threads_started":1}
I20260927 22:47:39.860289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5859
I20260927 22:47:39.861289 5920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:39.957221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5859
2026-09-27T22:47:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (711 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (386 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (389 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:40 dist-test-slave-kr25 krb5kdc[5969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:40 dist-test-slave-kr25 krb5kdc[5969](info): set up 2 sockets
Sep 27 22:47:40 dist-test-slave-kr25 krb5kdc[5969](info): commencing operation
krb5kdc: starting...
W20260927 22:47:41.720000 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:41 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:41.900844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39535
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:42.211802 5982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:42.212081 5982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:42.212176 5982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:42.219202 5982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:42.219303 5982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:42.219354 5982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:42.219408 5982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:42.219476 5982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:42.229351 5982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:42.230549 5982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:42.231788 5982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:42.242488 5987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:42.242468 5990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:42.242635 5988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:42.243522 5982 server_base.cc:1034] running on GCE node
I20260927 22:47:42.244212 5982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:42.245455 5982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:42.246654 5982 hybrid_clock.cc:648] HybridClock initialized: now 1790549262246610 us; error 29 us; skew 500 ppm
I20260927 22:47:42.246946 5982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:42 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:42.249842 5982 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:47:42.251159 5982 webserver.cc:460] Webserver started at http://127.0.97.62:40385/ using document root <none> and password file <none>
I20260927 22:47:42.251837 5982 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:42.251955 5982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:42.252238 5982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:42.254946 5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "8fb180e04bc34927904dbcdbbcf5dfd0"
format_stamp: "Formatted at 2026-09-27 22:47:42 on dist-test-slave-kr25"
server_key: "244dacb6fb61150b14e8c4069baa817a"
server_key_iv: "6ffbc89275a1543ce3f0d04cc8da13b5"
server_key_version: "encryptionkey@0"
I20260927 22:47:42.255710 5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "8fb180e04bc34927904dbcdbbcf5dfd0"
format_stamp: "Formatted at 2026-09-27 22:47:42 on dist-test-slave-kr25"
server_key: "244dacb6fb61150b14e8c4069baa817a"
server_key_iv: "6ffbc89275a1543ce3f0d04cc8da13b5"
server_key_version: "encryptionkey@0"
I20260927 22:47:42.260458 5982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:47:42.263782 5997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:42.265045 5982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:47:42.265213 5982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "8fb180e04bc34927904dbcdbbcf5dfd0"
format_stamp: "Formatted at 2026-09-27 22:47:42 on dist-test-slave-kr25"
server_key: "244dacb6fb61150b14e8c4069baa817a"
server_key_iv: "6ffbc89275a1543ce3f0d04cc8da13b5"
server_key_version: "encryptionkey@0"
I20260927 22:47:42.265375 5982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:42.310887 5982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:42.314949 5982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:42.315205 5982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:42.331903 5982 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39535
I20260927 22:47:42.331943 6048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39535 every 8 connection(s)
I20260927 22:47:42.333180 5982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:42.336732 6049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:42.338449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 5982
I20260927 22:47:42.338654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:42.339416 388 external_mini_cluster.cc:1428] Setting key 0e67869cd14b3f213ec2ee2cb180ab50
I20260927 22:47:42.348708 6049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0: Bootstrap starting.
I20260927 22:47:42.352392 6049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0: Neither blocks nor log segments found. Creating new log.
Sep 27 22:47:42 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:42.353541 6049 log.cc:824] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:42.356683 6049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0: No bootstrap required, opened a new log
I20260927 22:47:42.362003 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:42.341458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.97.62:39535)
0927 22:47:42.342162 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:42.342174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:42.342214 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:47:42.343894 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:42.343916 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:42.344319 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:42.344640 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:42.346216 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:42.347549 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:42.349165 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:42.349821 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:42.349937 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:42.354381 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:42.354388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:42.354405 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:42.354441 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:42.357889 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:42.358510 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:42.358519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:42.358531 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:42.358723 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:42.359026 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:42.359031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:42.359034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:42.359725 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:42.360024 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:42.360303 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 22:47:42.360713 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":302,"threads_started":1}
I20260927 22:47:42.365962 6049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39535 } }
I20260927 22:47:42.366261 6049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:42.366338 6049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb180e04bc34927904dbcdbbcf5dfd0, State: Initialized, Role: FOLLOWER
I20260927 22:47:42.366788 6049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39535 } }
I20260927 22:47:42.366968 6049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:42.367057 6049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:42.367163 6049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:42.369259 6049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39535 } }
I20260927 22:47:42.369699 6049 leader_election.cc:302] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fb180e04bc34927904dbcdbbcf5dfd0; no voters:
I20260927 22:47:42.370199 6049 leader_election.cc:288] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:42.370359 6054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:42.371484 6054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [term 1 LEADER]: Becoming Leader. State: Replica: 8fb180e04bc34927904dbcdbbcf5dfd0, State: Running, Role: LEADER
I20260927 22:47:42.371944 6054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39535 } }
I20260927 22:47:42.372406 6049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:42.375126 6055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39535 } } }
I20260927 22:47:42.375480 6055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:42.376233 6056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb180e04bc34927904dbcdbbcf5dfd0. Latest consensus state: current_term: 1 leader_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb180e04bc34927904dbcdbbcf5dfd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39535 } } }
I20260927 22:47:42.376509 6056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:42.377734 6063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:42.382570 6063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:42.390796 6063 catalog_manager.cc:1347] Generated new cluster ID: a300d110aafa44e2b15d65b5ad23457a
I20260927 22:47:42.390899 6063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:42.410670 6063 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:42.412686 6063 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:42.431916 6063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fb180e04bc34927904dbcdbbcf5dfd0: Generated new TSK 0
I20260927 22:47:42.432768 6063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:47:42.477145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39535
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:42.787504 6075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:42.787786 6075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:42.787900 6075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:42.795076 6075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:42.795184 6075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:42.795326 6075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:42.804582 6075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:42.805742 6075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:42.806959 6075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:42.818933 6080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:42.819003 6083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:42.819025 6081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:42.819710 6075 server_base.cc:1034] running on GCE node
I20260927 22:47:42.820289 6075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:42.821182 6075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:42.822383 6075 hybrid_clock.cc:648] HybridClock initialized: now 1790549262822333 us; error 37 us; skew 500 ppm
I20260927 22:47:42.822711 6075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:42 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:42.825785 6075 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:47:42.827092 6075 webserver.cc:460] Webserver started at http://127.0.97.1:35879/ using document root <none> and password file <none>
I20260927 22:47:42.827764 6075 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:42.827862 6075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:42.828091 6075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:42.830734 6075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "acc2359a1a5840a5a8123a880f0312d5"
format_stamp: "Formatted at 2026-09-27 22:47:42 on dist-test-slave-kr25"
server_key: "0414d19f1a98a861c96acec06041da2f"
server_key_iv: "227d0b7618213fc8a7883385e1855c62"
server_key_version: "encryptionkey@0"
I20260927 22:47:42.831403 6075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "acc2359a1a5840a5a8123a880f0312d5"
format_stamp: "Formatted at 2026-09-27 22:47:42 on dist-test-slave-kr25"
server_key: "0414d19f1a98a861c96acec06041da2f"
server_key_iv: "227d0b7618213fc8a7883385e1855c62"
server_key_version: "encryptionkey@0"
I20260927 22:47:42.836014 6075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:47:42.839043 6090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:42.840267 6075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:42.840389 6075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "acc2359a1a5840a5a8123a880f0312d5"
format_stamp: "Formatted at 2026-09-27 22:47:42 on dist-test-slave-kr25"
server_key: "0414d19f1a98a861c96acec06041da2f"
server_key_iv: "227d0b7618213fc8a7883385e1855c62"
server_key_version: "encryptionkey@0"
I20260927 22:47:42.840531 6075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:42.855012 6075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:42.858831 6075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:42.859042 6075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:42.859951 6075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:42.861173 6075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:42.861249 6075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:42.861361 6075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:42.861413 6075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:42.889430 6075 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40567
I20260927 22:47:42.889477 6202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40567 every 8 connection(s)
I20260927 22:47:42.890605 6075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:42.896670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 6075
I20260927 22:47:42.896855 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:42.897451 388 external_mini_cluster.cc:1428] Setting key 2e3efbb530b2824be340e4ea4a6bf005
Sep 27 22:47:42 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:42.908612 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:42.892877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48743 (local address 127.0.97.62:39535)
0927 22:47:42.893363 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:42.893369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:42.894340 (+ 971us) server_negotiation.cc:407] Connection header received
0927 22:47:42.896038 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:42.896042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:42.896105 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:42.896200 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:42.897622 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:42.898415 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:42.899932 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:42.900323 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:42.900429 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:42.903594 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:42.903599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:42.903602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:42.903639 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:42.905875 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:42.906513 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:42.906516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:42.906519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:42.906623 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:42.907004 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:42.907008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:42.907010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:42.907262 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:42.907383 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:42.908063 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 22:47:42.908267 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":261,"threads_started":1}
I20260927 22:47:42.909508 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:42.893495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39535 (local address 127.0.97.1:48743)
0927 22:47:42.894190 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:42.894231 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:42.895805 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:42.896383 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:42.896391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:42.896880 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:42.897456 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:42.897473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:42.898562 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:42.898566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:42.899808 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:42.899819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:42.900520 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:42.900527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:42.900735 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:42.901335 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:42.901356 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:42.903377 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:42.905994 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:42.906002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:42.906018 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:42.906400 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:42.906726 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:42.906731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:42.906734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:42.906900 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:42.907375 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:42.907382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:42.907726 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 22:47:42.907965 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 22:47:42.908449 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
I20260927 22:47:42.910926 6203 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39535
I20260927 22:47:42.911276 6203 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:42.912184 6203 heartbeater.cc:511] Master 127.0.97.62:39535 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:42.914567 6014 ts_manager.cc:194] Registered new tserver with Master: acc2359a1a5840a5a8123a880f0312d5 (127.0.97.1:40567)
I20260927 22:47:42.916015 6014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:47:42.938486 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39535
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:43.240659 6209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:43.240923 6209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:43.241039 6209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:43.248077 6209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:43.248173 6209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:43.248311 6209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:43.257753 6209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:43.259012 6209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:43.260424 6209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:43.272594 6217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:43.272686 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:43.272641 6215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:43.273432 6209 server_base.cc:1034] running on GCE node
I20260927 22:47:43.273993 6209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:43.274861 6209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:43.276044 6209 hybrid_clock.cc:648] HybridClock initialized: now 1790549263275995 us; error 42 us; skew 500 ppm
I20260927 22:47:43.276337 6209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.279261 6209 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:47:43.280658 6209 webserver.cc:460] Webserver started at http://127.0.97.2:45871/ using document root <none> and password file <none>
I20260927 22:47:43.281291 6209 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:43.281404 6209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:43.281677 6209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:43.284524 6209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "98a84510b53844a2bc958d448945387a"
format_stamp: "Formatted at 2026-09-27 22:47:43 on dist-test-slave-kr25"
server_key: "7b9b62d6fcd91abcc10667d620ee47a1"
server_key_iv: "d2e586700bcef1a3264accf36a34e608"
server_key_version: "encryptionkey@0"
I20260927 22:47:43.285254 6209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "98a84510b53844a2bc958d448945387a"
format_stamp: "Formatted at 2026-09-27 22:47:43 on dist-test-slave-kr25"
server_key: "7b9b62d6fcd91abcc10667d620ee47a1"
server_key_iv: "d2e586700bcef1a3264accf36a34e608"
server_key_version: "encryptionkey@0"
I20260927 22:47:43.290057 6209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:47:43.293279 6224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:43.294625 6209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:43.294766 6209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "98a84510b53844a2bc958d448945387a"
format_stamp: "Formatted at 2026-09-27 22:47:43 on dist-test-slave-kr25"
server_key: "7b9b62d6fcd91abcc10667d620ee47a1"
server_key_iv: "d2e586700bcef1a3264accf36a34e608"
server_key_version: "encryptionkey@0"
I20260927 22:47:43.294950 6209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:43.323494 6209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:43.327410 6209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:43.327699 6209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:43.328613 6209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:43.329878 6209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:43.329952 6209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:43.330062 6209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:43.330111 6209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:43.362993 6209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36659
I20260927 22:47:43.363023 6336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36659 every 8 connection(s)
I20260927 22:47:43.364318 6209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:43.364917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 6209
I20260927 22:47:43.365111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:43.365670 388 external_mini_cluster.cc:1428] Setting key 51b148fcd6f33096eb2c4dfc0ac46d8b
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:47:43.384634 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.366918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39591 (local address 127.0.97.62:39535)
0927 22:47:43.367063 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.367068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.368460 (+ 1392us) server_negotiation.cc:407] Connection header received
0927 22:47:43.370286 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.370291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.370384 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.370464 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.371971 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.372655 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.374279 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.374972 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.375083 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.378492 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:43.378498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:43.378502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:43.378539 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:43.381721 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.382398 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.382404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.382408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.382563 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.382943 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.382948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.382951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.383291 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:43.383471 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:43.384129 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 22:47:43.384375 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:47:43.385591 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.367577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39535 (local address 127.0.97.2:39591)
0927 22:47:43.368326 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:43.368370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:43.370043 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:43.370691 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:43.370699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:43.371171 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:43.371820 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:43.371838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.372828 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.372834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:43.374128 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:43.374144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.375151 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.375156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:43.375368 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.376065 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:43.376087 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:43.378262 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:43.381857 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:43.381864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:43.381877 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:43.382269 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:43.382651 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:43.382654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:43.382656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:43.382822 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:43.383416 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:43.383424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:43.383775 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 22:47:43.383994 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:47:43.384412 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":339,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:47:43.387105 6337 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39535
I20260927 22:47:43.387374 6337 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:43.388083 6337 heartbeater.cc:511] Master 127.0.97.62:39535 requested a full tablet report, sending...
I20260927 22:47:43.389436 6014 ts_manager.cc:194] Registered new tserver with Master: 98a84510b53844a2bc958d448945387a (127.0.97.2:36659)
I20260927 22:47:43.390391 6014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:47:43.408205 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39535
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:43.711077 6342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:43.711349 6342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:43.711501 6342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:43.718423 6342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:43.718531 6342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:43.718731 6342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:43.727807 6342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:43.728973 6342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:43.730192 6342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:43.742067 6348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:43.742024 6347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:43.742159 6350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:43.743040 6342 server_base.cc:1034] running on GCE node
I20260927 22:47:43.743637 6342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:43.744465 6342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:43.745658 6342 hybrid_clock.cc:648] HybridClock initialized: now 1790549263745614 us; error 36 us; skew 500 ppm
I20260927 22:47:43.745949 6342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.748870 6342 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:47:43.750195 6342 webserver.cc:460] Webserver started at http://127.0.97.3:34375/ using document root <none> and password file <none>
I20260927 22:47:43.750797 6342 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:43.750890 6342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:43.751121 6342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:43.753897 6342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "1b76b392fc1d4b98ade93d0ee4e8a335"
format_stamp: "Formatted at 2026-09-27 22:47:43 on dist-test-slave-kr25"
server_key: "43fb29cd3338c555461495476b96b410"
server_key_iv: "684a8cc0c1c17529a6e4da22eec64496"
server_key_version: "encryptionkey@0"
I20260927 22:47:43.754573 6342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1b76b392fc1d4b98ade93d0ee4e8a335"
format_stamp: "Formatted at 2026-09-27 22:47:43 on dist-test-slave-kr25"
server_key: "43fb29cd3338c555461495476b96b410"
server_key_iv: "684a8cc0c1c17529a6e4da22eec64496"
server_key_version: "encryptionkey@0"
I20260927 22:47:43.759239 6342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:47:43.762468 6357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:43.763731 6342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:47:43.763880 6342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "1b76b392fc1d4b98ade93d0ee4e8a335"
format_stamp: "Formatted at 2026-09-27 22:47:43 on dist-test-slave-kr25"
server_key: "43fb29cd3338c555461495476b96b410"
server_key_iv: "684a8cc0c1c17529a6e4da22eec64496"
server_key_version: "encryptionkey@0"
I20260927 22:47:43.764055 6342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:43.782727 6342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:43.786387 6342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:43.786651 6342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:43.787575 6342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:43.788865 6342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:43.788940 6342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:43.789062 6342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:43.789112 6342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:43.817956 6342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38569
I20260927 22:47:43.817999 6469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38569 every 8 connection(s)
I20260927 22:47:43.819164 6342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:43.823480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 6342
I20260927 22:47:43.823628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:43.824023 388 external_mini_cluster.cc:1428] Setting key 69d103e71912ef7f6c3ebf6d41bc9e3a
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:47:43.836961 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.821583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42865 (local address 127.0.97.62:39535)
0927 22:47:43.821759 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.821763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.822922 (+ 1159us) server_negotiation.cc:407] Connection header received
0927 22:47:43.824780 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.824786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.824870 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.824973 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.826705 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.827427 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.828874 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.829242 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.829329 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.832371 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:43.832376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:43.832379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:43.832408 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:43.834521 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.835122 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.835126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.835128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.835230 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.835598 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.835601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.835603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.835913 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:43.836018 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:43.836576 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 22:47:43.836720 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:47:43.838059 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.822073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39535 (local address 127.0.97.3:42865)
0927 22:47:43.822769 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:43.822816 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:43.824496 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:43.825229 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:43.825240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:43.825964 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:43.826552 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:43.826577 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.827570 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.827573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:43.828769 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:43.828781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.829403 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.829407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:43.829596 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.830163 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:43.830188 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:43.832192 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:43.834633 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:43.834639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:43.834650 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:43.835039 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:43.835304 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:43.835307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:43.835309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:43.835510 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:43.836008 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:43.836013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:43.836296 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 22:47:43.836504 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 22:47:43.836875 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":288,"threads_started":1}
I20260927 22:47:43.839416 6470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39535
I20260927 22:47:43.839728 6470 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:43.840432 6470 heartbeater.cc:511] Master 127.0.97.62:39535 requested a full tablet report, sending...
I20260927 22:47:43.841701 6013 ts_manager.cc:194] Registered new tserver with Master: 1b76b392fc1d4b98ade93d0ee4e8a335 (127.0.97.3:38569)
I20260927 22:47:43.842483 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42865
I20260927 22:47:43.850834 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/test-admin.keytab.
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.874228 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 22:47:43.878750 6477 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:43.876654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.0.97.1:40567)
0927 22:47:43.877175 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.877184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.877213 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:47:43.877359 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.877366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.877575 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.877723 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.878450 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52236: BlockingRecv error: recv got EOF from 127.0.0.1:52236 (error 108)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
W20260927 22:47:43.879134 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc)
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.880555 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.881747 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:43.896685 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.883139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.0.97.1:40567)
0927 22:47:43.883306 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.883311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.883571 (+ 260us) server_negotiation.cc:407] Connection header received
0927 22:47:43.883703 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.883706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.883775 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.883842 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.884724 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.885706 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.886949 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.887520 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.887613 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.891402 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:43.891412 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:43.891424 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:43.891491 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:43.894206 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.894919 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.894925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.894927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.895040 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.895369 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.895371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.895373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.895730 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:43.895961 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:43.896226 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:47:43.896447 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:47:43.901880 6477 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:43.900442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.0.97.1:40567)
0927 22:47:43.900615 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.900620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.900775 (+ 155us) server_negotiation.cc:407] Connection header received
0927 22:47:43.900922 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.900926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.900989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.901064 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.901701 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52262: BlockingRecv error: recv got EOF from 127.0.0.1:52262 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:47:43.902051 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.903453 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.904597 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:43.916742 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.905717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.0.97.1:40567)
0927 22:47:43.905902 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.905907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.906132 (+ 225us) server_negotiation.cc:407] Connection header received
0927 22:47:43.906235 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.906238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.906287 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.906359 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.907130 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.907889 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.908978 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.909375 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.909457 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.912545 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:43.912549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:43.912552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:43.912579 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:43.914737 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.915288 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.915291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.915293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.915391 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.915714 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.915717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.915719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.915966 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:43.916133 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:43.916341 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:47:43.916490 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:47:43.918704 6203 heartbeater.cc:503] Master 127.0.97.62:39535 was elected leader, sending a full tablet report...
W20260927 22:47:43.920564 6477 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:43.919025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.0.97.1:40567)
0927 22:47:43.919182 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.919188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.919348 (+ 160us) server_negotiation.cc:407] Connection header received
0927 22:47:43.919498 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.919503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.919591 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.919700 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.920370 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52278: BlockingRecv error: recv got EOF from 127.0.0.1:52278 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 22:47:43.920682 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549216186161-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.921952 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:43.923038 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:47:43 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:47:43.935030 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.924173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.0.97.1:40567)
0927 22:47:43.924352 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.924357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.924529 (+ 172us) server_negotiation.cc:407] Connection header received
0927 22:47:43.924627 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.924630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.924677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.924747 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.925584 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.926256 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.927368 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.927779 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.927874 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.930817 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:43.930821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:43.930824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:43.930853 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:43.933057 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.933599 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.933602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.933604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.933730 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.933987 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.933990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.933992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.934227 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:43.934432 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:43.934569 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 22:47:43.934728 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:43.936388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 6075
I20260927 22:47:43.946537 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:43.937000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.0.97.1:40567)
0927 22:47:43.937121 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:43.937125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:43.937481 (+ 356us) server_negotiation.cc:407] Connection header received
0927 22:47:43.937589 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:43.937593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:43.937640 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:43.937709 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:43.938485 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.939121 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.940266 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:43.940675 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:43.940767 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:43.942402 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:43.942406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:43.942409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:43.942435 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:43.944648 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.945156 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.945159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.945161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.945283 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:43.945537 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:43.945539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:43.945541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:43.945782 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:43.945952 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:43.946157 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:47:43.946316 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 22:47:43.947018 6198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:44.027334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 6075
I20260927 22:47:44.039902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 6209
Sep 27 22:47:44 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:47:44.054013 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:44.040707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.0.97.2:36659)
0927 22:47:44.041240 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:44.041247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:44.041270 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:47:44.041326 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:44.041332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:44.041534 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:44.041669 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:44.042599 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:44.043619 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:44.044811 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:44.045354 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:44.045471 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:44.049020 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:44.049031 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:44.049045 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:44.049074 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:44.051509 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:44.052102 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:44.052109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:44.052112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:44.052229 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:44.052560 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:44.052565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:44.052567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:44.052879 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:44.053141 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:44.053474 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 22:47:44.053712 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":239,"threads_started":1}
I20260927 22:47:44.055188 6332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:44.134234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 6209
I20260927 22:47:44.146579 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 6342
Sep 27 22:47:44 dist-test-slave-kr25 krb5kdc[5969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:47:44.160746 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:44.147413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.0.97.3:38569)
0927 22:47:44.148021 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:44.148028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:44.148053 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:47:44.148111 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:44.148118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:44.148324 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:44.148466 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:44.149458 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:44.150457 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:44.151720 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:44.152274 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:44.152385 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:44.155968 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:44.155978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:44.155993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:44.156028 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:44.158440 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:44.159033 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:44.159040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:44.159042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:44.159151 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:44.159409 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:44.159412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:44.159414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:44.159843 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:44.160070 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:44.160287 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 22:47:44.160472 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":290,"threads_started":1}
I20260927 22:47:44.161954 6465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:44.241374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 6342
I20260927 22:47:44.253252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 5982
I20260927 22:47:44.254304 6044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:44.352778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 5982
2026-09-27T22:47:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3625 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:44 dist-test-slave-kr25 krb5kdc[6497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:44 dist-test-slave-kr25 krb5kdc[6497](info): set up 2 sockets
Sep 27 22:47:44 dist-test-slave-kr25 krb5kdc[6497](info): commencing operation
krb5kdc: starting...
W20260927 22:47:45.347838 388 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:45 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 22:47:45.532301 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41249
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:45.834780 6510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:45.835067 6510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:45.835206 6510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:45.842272 6510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:45.842382 6510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:45.842460 6510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:45.842530 6510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:45.842589 6510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:45.851984 6510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:45.853181 6510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:45.854430 6510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:45.865509 6515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:45.865485 6516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:45.865849 6518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:45.866485 6510 server_base.cc:1034] running on GCE node
I20260927 22:47:45.867225 6510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:45.868610 6510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:45.869815 6510 hybrid_clock.cc:648] HybridClock initialized: now 1790549265869768 us; error 39 us; skew 500 ppm
I20260927 22:47:45.870133 6510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:45 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:45.877396 6510 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 22:47:45.878752 6510 webserver.cc:460] Webserver started at http://127.0.97.62:35683/ using document root <none> and password file <none>
I20260927 22:47:45.879391 6510 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:45.879527 6510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:45.879808 6510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:45.882568 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "625e9c22217f417a926a6fa02dbf95a1"
format_stamp: "Formatted at 2026-09-27 22:47:45 on dist-test-slave-kr25"
server_key: "19a9370f69ebc73451e76b44fb419241"
server_key_iv: "0fbe032174edc95ebb6b06d07fe318a0"
server_key_version: "encryptionkey@0"
I20260927 22:47:45.883258 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "625e9c22217f417a926a6fa02dbf95a1"
format_stamp: "Formatted at 2026-09-27 22:47:45 on dist-test-slave-kr25"
server_key: "19a9370f69ebc73451e76b44fb419241"
server_key_iv: "0fbe032174edc95ebb6b06d07fe318a0"
server_key_version: "encryptionkey@0"
I20260927 22:47:45.888106 6510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:47:45.891294 6525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:45.892620 6510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:47:45.892776 6510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "625e9c22217f417a926a6fa02dbf95a1"
format_stamp: "Formatted at 2026-09-27 22:47:45 on dist-test-slave-kr25"
server_key: "19a9370f69ebc73451e76b44fb419241"
server_key_iv: "0fbe032174edc95ebb6b06d07fe318a0"
server_key_version: "encryptionkey@0"
I20260927 22:47:45.892961 6510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:45.915884 6510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:45.919596 6510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:45.919845 6510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:45.936175 6510 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41249
I20260927 22:47:45.936221 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41249 every 8 connection(s)
I20260927 22:47:45.937420 6510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:45.938831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 6510
I20260927 22:47:45.939025 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:45.939841 388 external_mini_cluster.cc:1428] Setting key 33831d2543c1ed1e7bcd416ed16bb86b
I20260927 22:47:45.941829 6577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:47:45 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:45.954378 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: Bootstrap starting.
I20260927 22:47:45.957413 6577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:45.958560 6577 log.cc:824] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:45.960917 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:45.941742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.0.97.62:41249)
0927 22:47:45.942465 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:45.942475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:45.942514 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:47:45.944087 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:45.944108 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:45.944484 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:45.944825 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:45.946406 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:45.947739 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:45.949345 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:45.949966 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:45.950083 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:45.954410 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:45.954418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:45.954431 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:45.954470 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:45.957248 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:45.957950 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:45.957956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:45.957962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:45.958111 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:45.958506 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:45.958509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:45.958511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:45.959033 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:45.959300 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:45.959707 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 22:47:45.960018 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":323,"threads_started":1}
I20260927 22:47:45.961304 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: No bootstrap required, opened a new log
I20260927 22:47:45.968235 6577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } }
I20260927 22:47:45.968536 6577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:45.968616 6577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625e9c22217f417a926a6fa02dbf95a1, State: Initialized, Role: FOLLOWER
I20260927 22:47:45.969194 6577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } }
I20260927 22:47:45.969413 6577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:45.969578 6577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:45.969727 6577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:45.971659 6577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } }
I20260927 22:47:45.972074 6577 leader_election.cc:302] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625e9c22217f417a926a6fa02dbf95a1; no voters:
I20260927 22:47:45.972548 6577 leader_election.cc:288] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:45.972656 6582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:45.973235 6582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 1 LEADER]: Becoming Leader. State: Replica: 625e9c22217f417a926a6fa02dbf95a1, State: Running, Role: LEADER
I20260927 22:47:45.973789 6582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } }
I20260927 22:47:45.974306 6577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:45.976468 6583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } }
I20260927 22:47:45.976830 6583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:45.976758 6584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625e9c22217f417a926a6fa02dbf95a1. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } }
I20260927 22:47:45.977047 6584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:45.980062 6591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:45.984761 6591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:45.993424 6591 catalog_manager.cc:1347] Generated new cluster ID: 6cc4d210a841477cbc17b60811b2e259
I20260927 22:47:45.993531 6591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:46.002954 6591 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:46.004235 6591 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:46.013443 6591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: Generated new TSK 0
I20260927 22:47:46.014199 6591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 22:47:46.060861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41249
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:46.377808 6603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:46.378077 6603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:46.378201 6603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:46.385272 6603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:46.385377 6603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:46.385524 6603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:46.394982 6603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:46.396673 6603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:46.398093 6603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:46.410429 6608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:46.410444 6611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:46.410557 6609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:46.411164 6603 server_base.cc:1034] running on GCE node
I20260927 22:47:46.411759 6603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:46.412631 6603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:46.413856 6603 hybrid_clock.cc:648] HybridClock initialized: now 1790549266413801 us; error 46 us; skew 500 ppm
I20260927 22:47:46.414165 6603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:46 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:46.417378 6603 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:47:46.418910 6603 webserver.cc:460] Webserver started at http://127.0.97.1:38887/ using document root <none> and password file <none>
I20260927 22:47:46.419719 6603 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:46.419829 6603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:46.420075 6603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:46.422972 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "4b02ea733a314449a01b1db56ad1d2ff"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "4d603d025a0eaa24b0b352f4ebd05b09"
server_key_iv: "43e57adcbc3cadf89e79779321c15c64"
server_key_version: "encryptionkey@0"
I20260927 22:47:46.423786 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4b02ea733a314449a01b1db56ad1d2ff"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "4d603d025a0eaa24b0b352f4ebd05b09"
server_key_iv: "43e57adcbc3cadf89e79779321c15c64"
server_key_version: "encryptionkey@0"
I20260927 22:47:46.428897 6603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:47:46.432206 6618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:46.433498 6603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:47:46.433630 6603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "4b02ea733a314449a01b1db56ad1d2ff"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "4d603d025a0eaa24b0b352f4ebd05b09"
server_key_iv: "43e57adcbc3cadf89e79779321c15c64"
server_key_version: "encryptionkey@0"
I20260927 22:47:46.433806 6603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:46.462266 6603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:46.466444 6603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:46.466701 6603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:46.467671 6603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:46.468914 6603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:46.468983 6603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:46.469095 6603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:46.469148 6603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:46.498202 6603 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38275
I20260927 22:47:46.498271 6730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38275 every 8 connection(s)
I20260927 22:47:46.499527 6603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:46.501101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 6603
I20260927 22:47:46.501291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:46.501880 388 external_mini_cluster.cc:1428] Setting key 674a17287024800e9a9978dec1fa7123
Sep 27 22:47:46 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:47:46.522451 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:46.504021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37013 (local address 127.0.97.62:41249)
0927 22:47:46.507757 (+ 3736us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:46.507762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:46.507779 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:47:46.507985 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:46.507989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:46.508047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:46.508133 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:46.509970 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:46.510772 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:46.512384 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:46.512937 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:46.513084 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:46.516500 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:46.516506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:46.516510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:46.516545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:46.519613 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:46.520283 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:46.520289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:46.520292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:46.520449 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:46.520810 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:46.520815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:46.520818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:46.521146 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:46.521295 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:46.521903 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 22:47:46.522119 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3639,"thread_start_us":3546,"threads_started":1}
I20260927 22:47:46.523252 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:46.504726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.1:37013)
0927 22:47:46.505605 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:46.505655 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:46.507728 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:46.508385 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:46.508396 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:46.509058 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:46.509806 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:46.509828 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:46.510925 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:46.510928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:46.512242 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:46.512254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:46.513159 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:46.513165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:46.513383 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:46.514026 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:46.514047 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:46.516289 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:46.519743 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:46.519749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:46.519763 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:46.520151 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:46.520539 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:46.520541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:46.520543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:46.520707 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:46.521251 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:46.521257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:46.521550 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 22:47:46.521756 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 22:47:46.522154 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 22:47:46.524796 6731 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41249
I20260927 22:47:46.525216 6731 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:46.526039 6731 heartbeater.cc:511] Master 127.0.97.62:41249 requested a full tablet report, sending...
I20260927 22:47:46.528330 6542 ts_manager.cc:194] Registered new tserver with Master: 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:46.530078 6542 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 22:47:46.548411 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41249
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:46.850728 6737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:46.851039 6737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:46.851179 6737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:46.858273 6737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:46.858397 6737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:46.858553 6737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:46.869164 6737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:46.870450 6737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:46.871901 6737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:46.886605 6743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:46.886641 6742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:46.886679 6745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:46.887463 6737 server_base.cc:1034] running on GCE node
I20260927 22:47:46.888062 6737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:46.888962 6737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:46.890151 6737 hybrid_clock.cc:648] HybridClock initialized: now 1790549266890100 us; error 56 us; skew 500 ppm
I20260927 22:47:46.890444 6737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:46 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:46.893357 6737 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:47:46.894690 6737 webserver.cc:460] Webserver started at http://127.0.97.2:44075/ using document root <none> and password file <none>
I20260927 22:47:46.895334 6737 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:46.895468 6737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:46.895737 6737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:46.898492 6737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "e258531dd10243ecb9873b7f8c80091a"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "52d6da2e87e70a385949f415b798bc0d"
server_key_iv: "ff6ce4e2913d27e5976138c66955fad3"
server_key_version: "encryptionkey@0"
I20260927 22:47:46.899232 6737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e258531dd10243ecb9873b7f8c80091a"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "52d6da2e87e70a385949f415b798bc0d"
server_key_iv: "ff6ce4e2913d27e5976138c66955fad3"
server_key_version: "encryptionkey@0"
I20260927 22:47:46.904201 6737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:47:46.907609 6752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:46.908944 6737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:47:46.909090 6737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "e258531dd10243ecb9873b7f8c80091a"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "52d6da2e87e70a385949f415b798bc0d"
server_key_iv: "ff6ce4e2913d27e5976138c66955fad3"
server_key_version: "encryptionkey@0"
I20260927 22:47:46.909272 6737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:46.956375 6737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:46.960006 6737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:46.960258 6737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:46.961135 6737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:46.962355 6737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:46.962430 6737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:46.962545 6737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:46.962598 6737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:46.991128 6737 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38613
I20260927 22:47:46.991170 6864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38613 every 8 connection(s)
I20260927 22:47:46.992331 6737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:46.995035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 6737
I20260927 22:47:46.995187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:46.995693 388 external_mini_cluster.cc:1428] Setting key 78fcf004adcd20127363de3f9db29627
Sep 27 22:47:47 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:47.012266 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:46.994722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55799 (local address 127.0.97.62:41249)
0927 22:47:46.994919 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:46.994925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:46.996251 (+ 1326us) server_negotiation.cc:407] Connection header received
0927 22:47:46.998351 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:46.998356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:46.998430 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:46.998519 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:46.999964 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.000892 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.002753 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.003338 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.003480 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.006986 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:47.006991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:47.006994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:47.007026 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:47.009428 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.010106 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.010110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.010112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.010229 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.010617 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.010621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.010623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.010881 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:47.011005 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:47.011688 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 22:47:47.011897 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:47:47.013252 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:46.995238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:55799)
0927 22:47:46.996086 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:46.996136 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:46.998092 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:46.998737 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:46.998745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:46.999222 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:46.999819 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:46.999838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.001030 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.001035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.002607 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:47.002620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.003637 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.003646 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.003895 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.004549 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:47.004574 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:47.006771 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:47.009567 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.009576 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.009593 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.009986 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.010327 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.010332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.010336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.010509 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.010996 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:47.011004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:47.011319 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 22:47:47.011587 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 22:47:47.012101 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":296,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:47:47.014827 6865 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41249
I20260927 22:47:47.015120 6865 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:47.015851 6865 heartbeater.cc:511] Master 127.0.97.62:41249 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260927 22:47:47.017292 6542 ts_manager.cc:194] Registered new tserver with Master: e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613)
I20260927 22:47:47.018301 6542 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:47:47.038388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41249
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:47.341548 6870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:47.341830 6870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:47.341948 6870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:47.349092 6870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:47.349195 6870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:47.349330 6870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:47.358753 6870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:47.359982 6870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:47.361217 6870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:47.373237 6876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:47.373196 6878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:47.373696 6875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:47.374513 6870 server_base.cc:1034] running on GCE node
I20260927 22:47:47.375084 6870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:47.376076 6870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:47.377281 6870 hybrid_clock.cc:648] HybridClock initialized: now 1790549267377225 us; error 49 us; skew 500 ppm
I20260927 22:47:47.377568 6870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:47 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:47.380595 6870 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:47:47.381868 6870 webserver.cc:460] Webserver started at http://127.0.97.3:41129/ using document root <none> and password file <none>
I20260927 22:47:47.382488 6870 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:47.382583 6870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:47.382822 6870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:47.385620 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "abec420b66864c4db2a4ebb9074bd19f"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "7d84230a59b239046b1ab5aa5d5b0bb0"
server_key_iv: "bc62e12a7f942d316c76d6daac794a62"
server_key_version: "encryptionkey@0"
I20260927 22:47:47.386318 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "abec420b66864c4db2a4ebb9074bd19f"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "7d84230a59b239046b1ab5aa5d5b0bb0"
server_key_iv: "bc62e12a7f942d316c76d6daac794a62"
server_key_version: "encryptionkey@0"
I20260927 22:47:47.391201 6870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:47:47.394495 6885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:47.395896 6870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:47.396054 6870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "abec420b66864c4db2a4ebb9074bd19f"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "7d84230a59b239046b1ab5aa5d5b0bb0"
server_key_iv: "bc62e12a7f942d316c76d6daac794a62"
server_key_version: "encryptionkey@0"
I20260927 22:47:47.396245 6870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:47.411286 6870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:47.415891 6870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:47.416219 6870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:47.417315 6870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:47.418749 6870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:47.418838 6870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:47.418953 6870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:47.419006 6870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:47.448148 6870 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41383
I20260927 22:47:47.448192 6997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41383 every 8 connection(s)
I20260927 22:47:47.449319 6870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:47.454689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 6870
I20260927 22:47:47.454857 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:47.455331 388 external_mini_cluster.cc:1428] Setting key 57ae09207398132e41309f807771219a
Sep 27 22:47:47 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:47.467409 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:47.451769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50901 (local address 127.0.97.62:41249)
0927 22:47:47.451934 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:47.451938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:47.453117 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 22:47:47.454788 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:47.454794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:47.454881 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:47.454988 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:47.456572 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.457384 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.458984 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.459375 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.459504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.462609 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:47.462613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:47.462617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:47.462650 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:47.465079 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.465678 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.465681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.465683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.465786 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.466116 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.466118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.466120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.466363 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:47.466461 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:47.467020 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 22:47:47.467163 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:47:47.468372 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:47.452274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.3:50901)
0927 22:47:47.452962 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:47.453001 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:47.454524 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:47.455175 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:47.455183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:47.455730 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:47.456407 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:47.456434 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.457537 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.457541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.458841 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:47.458855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.459589 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.459596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.459822 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.460417 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:47.460439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:47.462426 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:47.465198 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.465204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.465216 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.465592 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.465869 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.465872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.465874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.466037 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.466449 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:47.466454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:47.466734 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 22:47:47.466952 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 22:47:47.467301 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":248,"threads_started":1}
I20260927 22:47:47.469622 6998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41249
I20260927 22:47:47.469904 6998 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:47.470595 6998 heartbeater.cc:511] Master 127.0.97.62:41249 requested a full tablet report, sending...
I20260927 22:47:47.471916 6541 ts_manager.cc:194] Registered new tserver with Master: abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383)
I20260927 22:47:47.472688 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50901
I20260927 22:47:47.482317 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 22:47:47.526818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39515
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41249,127.0.97.61:39515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:47:47.533217 6731 heartbeater.cc:503] Master 127.0.97.62:41249 was elected leader, sending a full tablet report...
W20260927 22:47:47.828439 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:47.828727 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:47.828840 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:47.836001 7003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:47.836102 7003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:47.836177 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:47.836251 7003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:47.836305 7003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:47.845834 7003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41249,127.0.97.61:39515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
W20260927 22:47:47.846163 7003 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:41249,127.0.97.61:39515'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 22:47:47.847146 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:47.848416 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:47.859220 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:47.859186 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:47.859249 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:47.860095 7003 server_base.cc:1034] running on GCE node
I20260927 22:47:47.860777 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:47.862037 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:47.863230 7003 hybrid_clock.cc:648] HybridClock initialized: now 1790549267863158 us; error 51 us; skew 500 ppm
I20260927 22:47:47.863559 7003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:47 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:47.866405 7003 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 22:47:47.867877 7003 webserver.cc:460] Webserver started at http://127.0.97.61:46373/ using document root <none> and password file <none>
I20260927 22:47:47.868530 7003 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:47.868626 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:47.868921 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:47.871724 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/instance:
uuid: "1b9ef0b13f3247bb942e425956a1518d"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "21ee5b7016a35cf6db22c812e937c555"
server_key_iv: "3550ec365180fbd978bf613ee2648e1e"
server_key_version: "encryptionkey@0"
I20260927 22:47:47.872434 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal/instance:
uuid: "1b9ef0b13f3247bb942e425956a1518d"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "21ee5b7016a35cf6db22c812e937c555"
server_key_iv: "3550ec365180fbd978bf613ee2648e1e"
server_key_version: "encryptionkey@0"
I20260927 22:47:47.877190 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 22:47:47.880252 7018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:47.881495 7003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 22:47:47.881644 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
uuid: "1b9ef0b13f3247bb942e425956a1518d"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "21ee5b7016a35cf6db22c812e937c555"
server_key_iv: "3550ec365180fbd978bf613ee2648e1e"
server_key_version: "encryptionkey@0"
I20260927 22:47:47.881817 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:47.905459 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:47.909452 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:47.909691 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:47.926141 7003 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39515
I20260927 22:47:47.926162 7069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39515 every 8 connection(s)
I20260927 22:47:47.927384 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:47:47.930756 7070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:47.933872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 7003
I20260927 22:47:47.938189 7070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:47:47 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:47:47 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:47.956096 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:47.941830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.0.97.62:41249)
0927 22:47:47.942015 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:47.942021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:47.942669 (+ 648us) server_negotiation.cc:407] Connection header received
0927 22:47:47.942970 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:47.942976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:47.943048 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:47.943140 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:47.944352 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.945079 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.946852 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.947249 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.947369 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.950828 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:47.950833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:47.950836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:47.950864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:47.953379 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.954043 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.954048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.954051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.954172 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.954558 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.954561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.954563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.954887 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:47.955015 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:47.955697 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 22:47:47.955892 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:47:47.957188 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:47.941750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:55598)
0927 22:47:47.942539 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:47.942585 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:47.942782 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:47.943217 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:47.943224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:47.943980 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:47.944209 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:47.944217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.945210 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.945214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.946731 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:47.946742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.947485 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.947493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.947636 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.948179 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:47.948199 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:47.950665 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:47.953487 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.953494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.953502 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.953949 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.954262 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.954266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.954269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.954458 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.955083 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:47.955090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:47.955307 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 22:47:47.956243 (+ 936us) client_negotiation.cc:240] Negotiation successful
0927 22:47:47.956459 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":402,"threads_started":1}
I20260927 22:47:47.957206 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:47.935996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.0.97.61:39515)
0927 22:47:47.936819 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:47.936834 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:47.936881 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:47:47.939085 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:47.939111 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:47.939651 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:47.940065 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:47.941759 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.943667 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.945397 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.946377 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.946553 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.950372 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:47.950383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:47.950398 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:47.950438 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:47.952994 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.953734 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.953741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.953749 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.953936 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.954302 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.954307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.954310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.954818 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:47.955089 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:47.955743 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 22:47:47.956128 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":356,"threads_started":1}
I20260927 22:47:47.958915 7070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:47:47 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:47:47.969158 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:47.959627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.0.1:35578)
0927 22:47:47.959767 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:47.959783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:47.959862 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:47.960087 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:47.960090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:47.960351 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:47.960480 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:47.960488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.961327 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.961330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.962142 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:47.962151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.962725 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.962729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:47.962842 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.963293 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:47.963309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:47.965005 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:47.967287 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.967291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.967293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.967659 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.967925 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:47.967928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:47.967930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:47.968035 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:47.968459 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:47.968463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:47.968611 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 22:47:47.968770 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:47:47.968946 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 22:47:47.969314 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:47.959566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.0.97.61:39515)
0927 22:47:47.959725 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:47.959734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:47.959837 (+ 103us) server_negotiation.cc:407] Connection header received
0927 22:47:47.959949 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:47.959952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:47.960002 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:47.960068 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:47.960588 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.961232 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.962244 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:47.962599 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:47.962683 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:47.965145 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:47.965149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:47.965152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:47.965178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:47.967186 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.967743 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.967746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.967748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.967851 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:47.968115 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:47.968118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:47.968120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:47.968368 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:47.968464 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:47.968884 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 22:47:47.969059 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:47:47.973450 7070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Bootstrap starting.
I20260927 22:47:47.976310 7070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:47.977243 7070 log.cc:824] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:47.979693 7070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: No bootstrap required, opened a new log
I20260927 22:47:47.986363 7070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } }
I20260927 22:47:47.986666 7070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:47.986755 7070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9ef0b13f3247bb942e425956a1518d, State: Initialized, Role: FOLLOWER
I20260927 22:47:47.987296 7070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } }
I20260927 22:47:47.988085 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } } }
I20260927 22:47:47.988384 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:47:47.989022 7070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:47:47.997383 7087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:47:47.997514 7087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:47:47.999691 7003 master_runner.cc:428] Detected that this master 1b9ef0b13f3247bb942e425956a1518d is joining an existing cluster
I20260927 22:47:47.999801 7003 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:39515
I20260927 22:47:48.020159 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.004691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.0.97.62:41249)
0927 22:47:48.004883 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.004889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.005751 (+ 862us) server_negotiation.cc:407] Connection header received
0927 22:47:48.005959 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.005964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.006036 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.006094 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.007226 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.008331 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.010176 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.010751 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.010841 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.013401 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.013407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.013413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.013450 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.016571 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.017959 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.017964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.017967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.018111 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.018513 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.018517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.018520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.018861 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.018976 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.019592 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.019812 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:47:48.020449 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.003908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.0.97.61:39515)
0927 22:47:48.004118 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.004124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.005024 (+ 900us) server_negotiation.cc:407] Connection header received
0927 22:47:48.005266 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.005270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.005366 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.005437 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.006921 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.007917 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.009652 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.010409 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.010526 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.013490 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.013496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.013500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.013531 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.017509 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.018297 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.018302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.018305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.018445 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.018820 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.018825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.018828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.019177 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.019315 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.019847 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.020047 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":54}
I20260927 22:47:48.021008 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.004295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:55604)
0927 22:47:48.005587 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.005604 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.005816 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.006238 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.006241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.006734 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.007058 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.007068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.008494 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.008499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.010012 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.010028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.011130 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.011136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.011317 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.011885 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.011908 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.013194 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.016709 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.016714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.016718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.017821 (+ 1103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.018239 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.018242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.018246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.018399 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.019070 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.019074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.019296 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.020333 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.020643 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"mutex_wait_us":54,"thread_start_us":566,"threads_started":1}
I20260927 22:47:48.021235 6865 heartbeater.cc:503] Master 127.0.97.62:41249 was elected leader, sending a full tablet report...
I20260927 22:47:48.021476 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.003908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.0.1:35588)
0927 22:47:48.004931 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.004950 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.005117 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.005624 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.005651 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.006329 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.006752 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.006765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.008071 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.008076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.009513 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.009526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.010778 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.010785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.010959 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.011890 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.011963 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.013303 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.017640 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.017646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.017649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.018160 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.018567 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.018571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.018575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.018714 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.019394 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.019398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.019574 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.020703 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.021010 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":23,"spinlock_wait_cycles":23808,"thread_start_us":1398,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:47:48.051671 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.033759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.0.97.61:39515)
0927 22:47:48.033918 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.033924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.036092 (+ 2168us) server_negotiation.cc:407] Connection header received
0927 22:47:48.036463 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.036468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.036553 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.036657 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.037862 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.039045 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.040692 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.041140 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.041249 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.045027 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.045037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.045042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.045080 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.048561 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.049408 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.049413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.049417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.049573 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.050007 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.050012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.050016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.050365 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.050511 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.051049 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.051279 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:47:48.052541 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.033758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.0.1:35594)
0927 22:47:48.035971 (+ 2213us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.036009 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.036229 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.036703 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.036709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.037465 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.037709 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.037720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.039175 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.039180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.040566 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.040578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.041280 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.041285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.041475 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.042078 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.042102 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.044818 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.048716 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.048729 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.048737 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.049275 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.049683 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.049688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.049691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.049880 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.050506 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.050514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.050711 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.050931 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.051258 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1973,"thread_start_us":1818,"threads_started":1}
I20260927 22:47:48.054569 6541 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:39515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 39515
}
attrs {
promote: true
}
}
dest_uuid: "625e9c22217f417a926a6fa02dbf95a1"
cas_config_opid_index: -1
I20260927 22:47:48.055496 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } }
I20260927 22:47:48.058519 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1b9ef0b13f3247bb942e425956a1518d (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } }
I20260927 22:47:48.059340 7045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:48.062667 7099 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:39515
I20260927 22:47:48.062889 7045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Refusing update from remote peer 625e9c22217f417a926a6fa02dbf95a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 22:47:48.063485 7003 master.cc:537] Master@127.0.97.61:39515 shutting down...
I20260927 22:47:48.063576 7099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } }
I20260927 22:47:48.063798 7099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:48.064366 7099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260927 22:47:48.066720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40505
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:48.067253 6529 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 22:47:48.070246 6529 consensus_peers.cc:487] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 -> Peer 1b9ef0b13f3247bb942e425956a1518d (127.0.97.61:39515): Couldn't send request to peer 1b9ef0b13f3247bb942e425956a1518d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 22:47:48.075407 7003 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:48.075855 7003 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:48.076026 7003 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: stopping tablet replica
I20260927 22:47:48.082177 7003 master.cc:552] Master@127.0.97.61:39515 shutdown complete.
I20260927 22:47:48.082296 7003 master_runner.cc:305] Clearing existing system tablet
I20260927 22:47:48.082774 7003 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:48.082912 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:48.085804 7003 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:47:48.087496 7105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:48.087827 7003 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:48.088225 7003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:47:48.088354 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
uuid: "1b9ef0b13f3247bb942e425956a1518d"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "21ee5b7016a35cf6db22c812e937c555"
server_key_iv: "3550ec365180fbd978bf613ee2648e1e"
server_key_version: "encryptionkey@0"
I20260927 22:47:48.093429 7003 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:48.100162 7003 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:47:48.100418 7003 log.cc:1196] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 22:47:48.100903 7003 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Deleting consensus metadata
I20260927 22:47:48.101195 7003 master_runner.cc:315] Copying system tablet from 127.0.97.62:41249
I20260927 22:47:48.108938 7003 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:41249
I20260927 22:47:48.133493 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.109801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.97.62:41249)
0927 22:47:48.109987 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.109993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.119971 (+ 9978us) server_negotiation.cc:407] Connection header received
0927 22:47:48.120120 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.120126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.120224 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.120344 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.121571 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.122672 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.124477 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.125109 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.125234 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.127389 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.127396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.127401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.127461 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.130662 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.131466 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.131472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.131476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.131609 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.132013 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.132018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.132021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.132349 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.132483 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.132912 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.133150 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:47:48.133560 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.109740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:55616)
0927 22:47:48.119833 (+ 10093us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.119864 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.120003 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.120438 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.120443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.121058 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.121394 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.121410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.122827 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.122837 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.124313 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.124330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.125275 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.125281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.125458 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.126174 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.126200 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.127177 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.130811 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.130816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.130820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.131318 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.131723 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.131728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.131731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.131908 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.132467 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.132472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.132643 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.132825 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.133133 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9952,"thread_start_us":9844,"threads_started":1}
I20260927 22:47:48.134752 6562 tablet_copy_service.cc:140] P 625e9c22217f417a926a6fa02dbf95a1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1b9ef0b13f3247bb942e425956a1518d ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55616)
I20260927 22:47:48.134997 6562 tablet_copy_service.cc:161] P 625e9c22217f417a926a6fa02dbf95a1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1b9ef0b13f3247bb942e425956a1518d at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55616: session id = 1b9ef0b13f3247bb942e425956a1518d-00000000000000000000000000000000
I20260927 22:47:48.139358 6562 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:47:48.141564 7003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:47:48.148275 7003 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: tablet copy: Starting download of 0 data blocks...
I20260927 22:47:48.148594 7003 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: tablet copy: Starting download of 1 WAL segments...
I20260927 22:47:48.151585 7003 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:47:48.155900 6562 tablet_copy_service.cc:342] P 625e9c22217f417a926a6fa02dbf95a1: Request end of tablet copy session 1b9ef0b13f3247bb942e425956a1518d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55616
I20260927 22:47:48.156116 6562 tablet_copy_service.cc:433] P 625e9c22217f417a926a6fa02dbf95a1: ending tablet copy session 1b9ef0b13f3247bb942e425956a1518d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1b9ef0b13f3247bb942e425956a1518d
I20260927 22:47:48.168198 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:48.170693 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:48.174862 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:48.182130 7115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:48.188828 7003 server_base.cc:1034] running on GCE node
I20260927 22:47:48.189246 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:48.189908 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:48.191200 7003 hybrid_clock.cc:648] HybridClock initialized: now 1790549268191138 us; error 34 us; skew 500 ppm
I20260927 22:47:48.191483 7003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:48.193666 7003 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 22:47:48.196398 7003 webserver.cc:460] Webserver started at http://127.0.97.61:32913/ using document root <none> and password file <none>
I20260927 22:47:48.196764 7003 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:48.196847 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:48.199980 7003 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:47:48.201807 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:48.202279 7003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:47:48.202409 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
uuid: "1b9ef0b13f3247bb942e425956a1518d"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "21ee5b7016a35cf6db22c812e937c555"
server_key_iv: "3550ec365180fbd978bf613ee2648e1e"
server_key_version: "encryptionkey@0"
I20260927 22:47:48.202560 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:48.221065 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:48.224931 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:48.248389 7003 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39515
I20260927 22:47:48.248458 7173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39515 every 8 connection(s)
I20260927 22:47:48.249402 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:47:48.254938 7174 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:47:48.258543 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Bootstrap starting.
I20260927 22:47:48.273257 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:47:48.273767 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Bootstrap complete.
I20260927 22:47:48.274828 7174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } }
I20260927 22:47:48.275116 7174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b9ef0b13f3247bb942e425956a1518d, State: Initialized, Role: LEARNER
I20260927 22:47:48.275388 7174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } }
I20260927 22:47:48.276525 7174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:48.276284 7177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } }
I20260927 22:47:48.276849 7177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: LEARNER
I20260927 22:47:48.287194 7188 catalog_manager.cc:1259] Loaded cluster ID: 6cc4d210a841477cbc17b60811b2e259
I20260927 22:47:48.287302 7188 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: loading cluster ID for follower catalog manager: success
I20260927 22:47:48.291049 7188 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: acquiring CA information for follower catalog manager: success
I20260927 22:47:48.293689 7188 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:47:48.420753 7103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:48.421056 7103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:48.421196 7103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:48.428517 7103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:48.428646 7103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:48.428723 7103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:48.428794 7103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:48.428860 7103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:48.439275 7103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:48.440663 7103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:48.441890 7103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:48.452772 7193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:48.453135 7194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:48.453299 7196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:48.453954 7103 server_base.cc:1034] running on GCE node
I20260927 22:47:48.454613 7103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:48.455969 7103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:48.457176 7103 hybrid_clock.cc:648] HybridClock initialized: now 1790549268457127 us; error 50 us; skew 500 ppm
I20260927 22:47:48.457471 7103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:48.460363 7103 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 22:47:48.461717 7103 webserver.cc:460] Webserver started at http://127.0.97.60:33889/ using document root <none> and password file <none>
I20260927 22:47:48.462354 7103 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:48.462458 7103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:48.462733 7103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:48.465683 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/instance:
uuid: "9e17792ade1a4f71bcb2d44f25e57d8e"
format_stamp: "Formatted at 2026-09-27 22:47:48 on dist-test-slave-kr25"
server_key: "f5c73e69a4f5d9cbf41ad23d35c18522"
server_key_iv: "49b8f540743ba558c742d29e6e824869"
server_key_version: "encryptionkey@0"
I20260927 22:47:48.466413 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal/instance:
uuid: "9e17792ade1a4f71bcb2d44f25e57d8e"
format_stamp: "Formatted at 2026-09-27 22:47:48 on dist-test-slave-kr25"
server_key: "f5c73e69a4f5d9cbf41ad23d35c18522"
server_key_iv: "49b8f540743ba558c742d29e6e824869"
server_key_version: "encryptionkey@0"
I20260927 22:47:48.471371 7103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:47:48.475116 6998 heartbeater.cc:503] Master 127.0.97.62:41249 was elected leader, sending a full tablet report...
I20260927 22:47:48.475162 7203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:48.477007 7103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:47:48.477306 7103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
uuid: "9e17792ade1a4f71bcb2d44f25e57d8e"
format_stamp: "Formatted at 2026-09-27 22:47:48 on dist-test-slave-kr25"
server_key: "f5c73e69a4f5d9cbf41ad23d35c18522"
server_key_iv: "49b8f540743ba558c742d29e6e824869"
server_key_version: "encryptionkey@0"
I20260927 22:47:48.477519 7103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:48.495656 7103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:48.500048 7103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:48.500335 7103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:48.518764 7103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40505
I20260927 22:47:48.518785 7254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40505 every 8 connection(s)
I20260927 22:47:48.520030 7103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:47:48.523502 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:48.529815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 7103
I20260927 22:47:48.529476 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:47:48.546309 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.539585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.0.1:35598)
0927 22:47:48.539834 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.539851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.539939 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.540861 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.540866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.540894 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:48.541096 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.541106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.542521 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.542524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.544309 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.544323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.545544 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.545550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.545770 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.545828 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.545906 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.546033 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 22:47:48.546893 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.539762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.0.97.61:39515)
0927 22:47:48.540537 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.540544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.540561 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:47:48.540617 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.540621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.540690 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.540799 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:48.541255 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.542390 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.544454 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.545368 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.546172 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.546388 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.546442 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.546570 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":513,"threads_started":1}
I20260927 22:47:48.548089 7149 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 22:47:48.548851 7177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625e9c22217f417a926a6fa02dbf95a1. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } }
I20260927 22:47:48.549124 7177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: LEARNER
I20260927 22:47:48.550802 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.532092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.0.97.62:41249)
0927 22:47:48.532293 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.532299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.533375 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 22:47:48.534979 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.534985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.535054 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.535142 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.536819 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.537644 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.539251 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.539940 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.540038 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.544373 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.544379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.544384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.544420 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.547479 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.548166 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.548172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.548176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.548293 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.548663 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.548668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.548671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.549450 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.549573 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.550228 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.550434 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:47:48.551749 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.532414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:55620)
0927 22:47:48.533223 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.533258 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.534713 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.535371 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.535378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.536152 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.536660 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.536683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.537783 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.537788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.539115 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.539126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.540166 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.540174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.540437 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.541399 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.541429 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.544175 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.547591 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.547601 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.547614 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.548062 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.548387 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.548392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.548396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.548568 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.549580 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.549588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.549901 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.550138 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.550606 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"spinlock_wait_cycles":13184,"thread_start_us":536,"threads_started":1}
I20260927 22:47:48.552099 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.532308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.0.97.60:40505)
0927 22:47:48.533034 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.533054 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.533098 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:47:48.534686 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.534707 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.534948 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.535143 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.536863 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.538189 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.539844 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.541083 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.541203 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.545553 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.545579 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.545595 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.545636 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.549053 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.549773 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.549781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.549784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.549911 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.550245 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.550250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.550252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.550570 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.550820 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.551586 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.551755 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
I20260927 22:47:48.553216 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:47:48.563953 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.553948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.0.1:35612)
0927 22:47:48.554101 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.554115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.554212 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.554507 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.554512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.554795 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.554915 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.554922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.555828 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.555831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.556800 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.556818 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.557384 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.557388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.557505 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.557969 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.557984 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.559788 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.562228 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.562232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.562234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.562563 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.562826 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.562829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.562831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.562938 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.563302 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.563305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.563399 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.563556 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.563756 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:47:48.563995 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.553867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.0.97.61:39515)
0927 22:47:48.553995 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.554000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.554171 (+ 171us) server_negotiation.cc:407] Connection header received
0927 22:47:48.554308 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.554313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.554381 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.554470 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.555021 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.555725 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.556916 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.557285 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.557369 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.559950 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.559955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.559958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.559987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.562125 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.562642 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.562645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.562648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.562754 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.563005 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.563008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.563010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.563228 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.563327 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.563617 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.563787 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:47:48.564750 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:47:48.575573 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.565337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.0.1:49078)
0927 22:47:48.565458 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.565472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.565549 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.565792 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.565796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.566079 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.566191 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.566198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.567100 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.567104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.567913 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.567924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.568519 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.568525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.568686 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.569085 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.569099 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.571348 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.573591 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.573595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.573599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.573905 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.574178 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.574182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.574185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.574312 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.574735 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.574739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.574854 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.574958 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.575178 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:47:48.575631 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.565482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.97.60:40505)
0927 22:47:48.565601 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.565605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.565619 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:47:48.565652 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.565655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.565698 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.565768 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.566302 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.566994 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.568023 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.568393 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.568506 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.571518 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.571522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.571525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.571549 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.573490 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.573997 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.574002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.574004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.574093 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.574400 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.574404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.574408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.574631 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.574748 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.575043 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.575325 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":37}
I20260927 22:47:48.580278 7255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Bootstrap starting.
I20260927 22:47:48.583364 7255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:48.584321 7255 log.cc:824] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:48.586655 7255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: No bootstrap required, opened a new log
I20260927 22:47:48.593750 7255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } }
I20260927 22:47:48.594056 7255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:48.594157 7255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e17792ade1a4f71bcb2d44f25e57d8e, State: Initialized, Role: FOLLOWER
I20260927 22:47:48.594666 7255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } }
I20260927 22:47:48.595408 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } } }
I20260927 22:47:48.595777 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:47:48.596426 7255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:47:48.604852 7273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:47:48.605026 7273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:47:48.606123 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 6603
I20260927 22:47:48.609534 7103 master_runner.cc:428] Detected that this master 9e17792ade1a4f71bcb2d44f25e57d8e is joining an existing cluster
I20260927 22:47:48.609640 7103 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:40505
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 22:47:48.628082 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.607559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.0.97.1:38275)
0927 22:47:48.608032 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.608040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.608068 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:47:48.608132 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.608140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.608387 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.609051 (+ 664us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.610265 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.611633 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.613050 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.613622 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.614308 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.619996 (+ 5688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.620009 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.620024 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.620063 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.624087 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.625002 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.625014 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.625018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.625161 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.625580 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.625584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.625587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.625987 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.626265 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.627417 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.627710 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":460,"threads_started":1}
I20260927 22:47:48.629689 6726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:48.638293 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.614466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.0.97.62:41249)
0927 22:47:48.614623 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.614630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.615944 (+ 1314us) server_negotiation.cc:407] Connection header received
0927 22:47:48.616168 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.616172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.616264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.616328 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.618132 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.619178 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.621167 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.621821 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.621912 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.630965 (+ 9053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.630972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.630977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.631017 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.634678 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.635485 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.635490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.635494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.635637 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.636295 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.636300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.636303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.636647 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.636772 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.637636 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.637861 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 22:47:48.639616 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.614022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:55630)
0927 22:47:48.615856 (+ 1834us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.615873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.616039 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.616514 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.616518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.617630 (+ 1112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.617966 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.617979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.619359 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.619364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.621012 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.621026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.627041 (+ 6015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.627047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.627228 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.627842 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.627864 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.628941 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.634820 (+ 5879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.634825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.634829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.635281 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.635961 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.635966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.635970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.636152 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.636909 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.636914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.637271 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.638846 (+ 1575us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.639141 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"thread_start_us":1577,"threads_started":1}
I20260927 22:47:48.640549 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.614143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.0.1:49092)
0927 22:47:48.621881 (+ 7738us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.621897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.621997 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.622376 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.622381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.622996 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.623333 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.623345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.625271 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.625276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.626810 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.626825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.630312 (+ 3487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.630318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.630485 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.633149 (+ 2664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.633175 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.634188 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.637976 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.638004 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.638008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.638458 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.638825 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.638828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.638832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.638978 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.639618 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.639622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.639790 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.639915 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.640180 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7603,"thread_start_us":7424,"threads_started":1}
I20260927 22:47:48.641360 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.614521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.97.60:40505)
0927 22:47:48.614780 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.614786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.622042 (+ 7256us) server_negotiation.cc:407] Connection header received
0927 22:47:48.622085 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.622089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.622153 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.622242 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.623511 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.625024 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.629141 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.629940 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.630062 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.634516 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.634522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.634527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.634560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.637857 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.638561 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.638566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.638570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.638704 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.639077 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.639082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.639084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.639392 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.639568 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.640841 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.641035 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":131}
I20260927 22:47:48.650586 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.620301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.97.61:39515)
0927 22:47:48.621044 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.621051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.625057 (+ 4006us) server_negotiation.cc:407] Connection header received
0927 22:47:48.625513 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.625519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.625596 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.625662 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.627085 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.628070 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.630001 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.630695 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.630791 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.644581 (+ 13790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.644588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.644592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.644634 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.647732 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.648517 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.648523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.648527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.648663 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.649088 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.649093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.649096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.649400 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.649507 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.650064 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.650273 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612}
I20260927 22:47:48.651742 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.620719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.0.1:35624)
0927 22:47:48.624977 (+ 4258us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.624992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.625351 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.625867 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.625871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.626537 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.626901 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.626914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.628256 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.628260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.629820 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.629834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.631083 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.631093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.631259 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.632150 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.632174 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.644361 (+ 12187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.647946 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.647950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.647953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.648385 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.648815 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.648819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.648823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.648970 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.649593 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.649598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.649794 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.651020 (+ 1226us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.651307 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4140,"mutex_wait_us":6,"thread_start_us":4353,"threads_started":1}
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:47:48.659688 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.645977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.0.1:49096)
0927 22:47:48.646331 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.646349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.646440 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.647106 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.647111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.647576 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.647762 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.647772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.649188 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.649194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.650456 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.650470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.651526 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.651533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.651719 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.652253 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.652273 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.654200 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.657569 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.657575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.657579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.657905 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.658199 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.658204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.658208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.658318 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.658793 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.658798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.658919 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.659042 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.659291 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260927 22:47:48.659729 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.646610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.0.97.60:40505)
0927 22:47:48.646759 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.646764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.646780 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:47:48.646856 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.646861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.646952 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.647878 (+ 926us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.648057 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.649056 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.650651 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.651348 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.654326 (+ 2978us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.654494 (+ 168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.654498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.654501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.654520 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.657450 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.658002 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.658007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.658010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.658106 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.658408 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.658413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.658417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.658657 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.658771 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.659165 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.659392 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:48.661127 6541 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:40505: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 40505
}
attrs {
promote: true
}
}
dest_uuid: "625e9c22217f417a926a6fa02dbf95a1"
cas_config_opid_index: 5
I20260927 22:47:48.661964 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } }
I20260927 22:47:48.670946 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.671510 7149 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 LEARNER]: Refusing update from remote peer 625e9c22217f417a926a6fa02dbf95a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:47:48.673142 7230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:48.677474 7230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Refusing update from remote peer 625e9c22217f417a926a6fa02dbf95a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:47:48.688680 7286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 22:47:48.690940 7099 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:40505
I20260927 22:47:48.691596 7103 master.cc:537] Master@127.0.97.60:40505 shutting down...
I20260927 22:47:48.696717 7286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:48.702544 7103 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:48.702984 7103 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:48.703245 7103 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: stopping tablet replica
I20260927 22:47:48.710371 7103 master.cc:552] Master@127.0.97.60:40505 shutdown complete.
I20260927 22:47:48.710490 7103 master_runner.cc:305] Clearing existing system tablet
I20260927 22:47:48.710932 7103 fs_manager.cc:359] Metadata directory not provided
W20260927 22:47:48.717338 6526 consensus_peers.cc:487] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 -> Peer 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60:40505): Couldn't send request to peer 9e17792ade1a4f71bcb2d44f25e57d8e. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:47:48.718570 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.719002 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:48.719096 7149 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.722114 7103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:48.722216 7177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: true } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.722455 7177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: LEARNER
I20260927 22:47:48.723366 7285 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: attempting to promote NON_VOTER 1b9ef0b13f3247bb942e425956a1518d to VOTER
I20260927 22:47:48.724197 7285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } }
I20260927 22:47:48.726976 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 LEARNER]: Refusing update from remote peer 625e9c22217f417a926a6fa02dbf95a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:47:48.727315 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.726868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.0.1:49100)
0927 22:47:48.727026 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.727139 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:47:48.728153 7286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 22:47:48.728405 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.728028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.0.1:49112)
0927 22:47:48.728157 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.728244 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 22:47:48.728746 6526 consensus_peers.cc:487] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 -> Peer 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60:40505): Couldn't send request to peer 9e17792ade1a4f71bcb2d44f25e57d8e. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:47:48.729542 7103 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 22:47:48.730777 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1b9ef0b13f3247bb942e425956a1518d (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.735149 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.735394 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:48.736547 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1b9ef0b13f3247bb942e425956a1518d (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.739722 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.740506 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:48.742268 7177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.742491 7177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:47:48.743474 7291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:48.743848 7103 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:48.744122 7103 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.008s
I20260927 22:47:48.744270 7103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
uuid: "9e17792ade1a4f71bcb2d44f25e57d8e"
format_stamp: "Formatted at 2026-09-27 22:47:48 on dist-test-slave-kr25"
server_key: "f5c73e69a4f5d9cbf41ad23d35c18522"
server_key_iv: "49b8f540743ba558c742d29e6e824869"
server_key_version: "encryptionkey@0"
I20260927 22:47:48.749158 7103 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:48.755856 7103 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:47:48.756003 7103 log.cc:1196] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 22:47:48.756451 7103 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Deleting consensus metadata
I20260927 22:47:48.756683 7103 master_runner.cc:315] Copying system tablet from 127.0.97.62:41249
I20260927 22:47:48.759460 7103 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:41249
I20260927 22:47:48.777741 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.760217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.0.97.62:41249)
0927 22:47:48.760414 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:48.760420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:48.763961 (+ 3541us) server_negotiation.cc:407] Connection header received
0927 22:47:48.764148 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:48.764154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:48.764235 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:48.764334 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:48.765263 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.766399 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.768230 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.768875 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.768989 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.771614 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:48.771620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:48.771626 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:48.771674 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:48.774914 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.775703 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.775709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.775712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.775850 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:48.776226 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:48.776236 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:48.776240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:48.776554 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:48.776699 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:48.777128 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 22:47:48.777396 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:47:48.777757 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:48.760143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:55632)
0927 22:47:48.763835 (+ 3692us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:48.763858 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:48.764007 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:48.764377 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:48.764382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:48.764873 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:48.765107 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.765122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.766554 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.766559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.768077 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:48.768092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:48.769027 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:48.769032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:48.769192 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:48.769885 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:48.769906 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:48.771404 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:48.775031 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.775037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.775041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.775582 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.775957 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:48.775962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:48.775966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:48.776122 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:48.776671 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:48.776676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:48.776872 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 22:47:48.776996 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:47:48.777344 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3559,"thread_start_us":3406,"threads_started":1}
I20260927 22:47:48.778692 6562 tablet_copy_service.cc:140] P 625e9c22217f417a926a6fa02dbf95a1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9e17792ade1a4f71bcb2d44f25e57d8e ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55632)
I20260927 22:47:48.778844 6562 tablet_copy_service.cc:161] P 625e9c22217f417a926a6fa02dbf95a1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9e17792ade1a4f71bcb2d44f25e57d8e at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55632: session id = 9e17792ade1a4f71bcb2d44f25e57d8e-00000000000000000000000000000000
I20260927 22:47:48.782337 6562 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:47:48.784125 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:47:48.787019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 6603
I20260927 22:47:48.789927 7103 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: tablet copy: Starting download of 0 data blocks...
I20260927 22:47:48.790197 7103 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: tablet copy: Starting download of 1 WAL segments...
I20260927 22:47:48.792388 7103 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:47:48.796555 6562 tablet_copy_service.cc:342] P 625e9c22217f417a926a6fa02dbf95a1: Request end of tablet copy session 9e17792ade1a4f71bcb2d44f25e57d8e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55632
I20260927 22:47:48.796764 6562 tablet_copy_service.cc:433] P 625e9c22217f417a926a6fa02dbf95a1: ending tablet copy session 9e17792ade1a4f71bcb2d44f25e57d8e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9e17792ade1a4f71bcb2d44f25e57d8e
I20260927 22:47:48.803370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38275
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41249
--webserver_port=38887
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:47:48.807529 7103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:48.811345 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:48.811836 7300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:48.824926 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:48.825150 7103 server_base.cc:1034] running on GCE node
I20260927 22:47:48.825456 7103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:48.826031 7103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:48.827275 7103 hybrid_clock.cc:648] HybridClock initialized: now 1790549268827232 us; error 35 us; skew 500 ppm
I20260927 22:47:48.827513 7103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:48 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:48.829173 7103 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 22:47:48.832198 7103 webserver.cc:460] Webserver started at http://127.0.97.60:45743/ using document root <none> and password file <none>
I20260927 22:47:48.832454 7103 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:48.832535 7103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:48.834741 7103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 22:47:48.836208 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:48.836573 7103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:47:48.836676 7103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
uuid: "9e17792ade1a4f71bcb2d44f25e57d8e"
format_stamp: "Formatted at 2026-09-27 22:47:48 on dist-test-slave-kr25"
server_key: "f5c73e69a4f5d9cbf41ad23d35c18522"
server_key_iv: "49b8f540743ba558c742d29e6e824869"
server_key_version: "encryptionkey@0"
I20260927 22:47:48.836826 7103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:48.845896 7103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:48.848798 7103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:48.866328 7103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40505
I20260927 22:47:48.866351 7360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40505 every 8 connection(s)
I20260927 22:47:48.867168 7103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:47:48.872663 7361 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:47:48.876492 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Bootstrap starting.
I20260927 22:47:48.893728 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:47:48.894205 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: Bootstrap complete.
I20260927 22:47:48.895222 7361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } }
I20260927 22:47:48.895547 7361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e17792ade1a4f71bcb2d44f25e57d8e, State: Initialized, Role: LEARNER
I20260927 22:47:48.895817 7361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } }
I20260927 22:47:48.897436 7361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:48.897190 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:48.898272 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: This master's current role is: LEARNER
I20260927 22:47:48.918728 7375 catalog_manager.cc:1259] Loaded cluster ID: 6cc4d210a841477cbc17b60811b2e259
I20260927 22:47:48.918915 7375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: loading cluster ID for follower catalog manager: success
I20260927 22:47:48.922340 7375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: acquiring CA information for follower catalog manager: success
I20260927 22:47:48.924520 7375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:47:49.150576 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:49.150866 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:49.150987 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:49.158167 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:49.158298 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:49.158452 7299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:49.168313 7299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38887
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:49.169792 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:49.171115 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:49.184350 7383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:49.184340 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:49.184379 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:49.184942 7299 server_base.cc:1034] running on GCE node
I20260927 22:47:49.185473 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:49.186285 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:49.187495 7299 hybrid_clock.cc:648] HybridClock initialized: now 1790549269187440 us; error 43 us; skew 500 ppm
I20260927 22:47:49.187788 7299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:49.190598 7299 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:47:49.191879 7299 webserver.cc:460] Webserver started at http://127.0.97.1:38887/ using document root <none> and password file <none>
I20260927 22:47:49.192499 7299 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:49.192615 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:49.197695 7299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:47:49.200757 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:49.202037 7299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:49.202170 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "4b02ea733a314449a01b1db56ad1d2ff"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "4d603d025a0eaa24b0b352f4ebd05b09"
server_key_iv: "43e57adcbc3cadf89e79779321c15c64"
server_key_version: "encryptionkey@0"
I20260927 22:47:49.203086 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:49.230567 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:49.235195 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:49.235469 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:49.236408 7299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:49.237897 7299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:49.237950 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.231124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.0.1:49120)
0927 22:47:49.231899 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:49.231916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:49.232036 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:49.232728 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:49.232733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:49.233025 (+ 292us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:49.233188 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.233198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.234526 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.234532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.235910 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.235922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.237167 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.237172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.237369 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.237395 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:47:49.237474 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:47:49.237609 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":584,"threads_started":1}
I20260927 22:47:49.237960 7396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.231262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.0.97.60:40505)
0927 22:47:49.232363 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.232370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.232388 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:49.232457 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.232462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.232552 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.232683 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:49.233329 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.234399 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.236037 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.237008 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.237124 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.237316 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.237508 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.237632 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":272,"threads_started":1}
I20260927 22:47:49.237998 7299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:49.238091 7299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:49.238152 7299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:49.238883 7336 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 22:47:49.239415 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625e9c22217f417a926a6fa02dbf95a1. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:49.239745 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: This master's current role is: LEARNER
I20260927 22:47:49.271109 7299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38275
I20260927 22:47:49.271163 7504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38275 every 8 connection(s)
I20260927 22:47:49.272452 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:49.277997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 7299
I20260927 22:47:49.278290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 6737
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:47:49.302377 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.276121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50489 (local address 127.0.97.62:41249)
0927 22:47:49.276454 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.276459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.277166 (+ 707us) server_negotiation.cc:407] Connection header received
0927 22:47:49.279728 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.279733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.279818 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.279883 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.282207 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.283290 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.286036 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.286682 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.286770 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.292548 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.292555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.292561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.292602 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.297160 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.298033 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.298047 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.298050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.298225 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.298756 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.298761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.298765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.299097 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.299514 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.301774 (+ 2260us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.302003 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260927 22:47:49.303797 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.275627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.1:50489)
0927 22:47:49.277029 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:49.277072 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:49.279422 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:49.280434 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:49.280447 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:49.281223 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:49.282017 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.282040 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.283787 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.283792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.285865 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.285880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.287075 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.287082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.287415 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.288252 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:49.288282 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:49.292286 (+ 4004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:49.297335 (+ 5049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.297344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.297361 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.297889 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.298356 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.298361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.298365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.298626 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.299584 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:49.299592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:49.300012 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 22:47:49.300627 (+ 615us) client_negotiation.cc:240] Negotiation successful
0927 22:47:49.301155 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"spinlock_wait_cycles":16256,"thread_start_us":17588,"threads_started":1}
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:47:49.305568 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.275627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.97.1:47493)
0927 22:47:49.286184 (+ 10557us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:49.286203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:49.286615 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:49.287134 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:49.287139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:49.287826 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:49.288117 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.288128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.289678 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.289684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.291499 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.291515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.292729 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.292735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.292939 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.293930 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:49.293949 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:49.298294 (+ 4345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:49.301917 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.301922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.301926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.302419 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.302976 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.302981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.302985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.303116 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.303977 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:49.303981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:49.304126 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 22:47:49.304873 (+ 747us) client_negotiation.cc:240] Negotiation successful
0927 22:47:49.305141 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10382,"thread_start_us":940,"threads_started":1}
I20260927 22:47:49.305606 7396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.275923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47493 (local address 127.0.97.60:40505)
0927 22:47:49.276316 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.276322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.286306 (+ 9984us) server_negotiation.cc:407] Connection header received
0927 22:47:49.286767 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.286774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.286859 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.286930 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.288305 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.289468 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.291668 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.292329 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.292433 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.298497 (+ 6064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.298503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.298508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.298555 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.301766 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.302646 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.302652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.302655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.302837 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.303236 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.303241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.303244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.303578 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.303895 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.305081 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.305298 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260927 22:47:49.306332 7506 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41249
I20260927 22:47:49.306671 7506 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:49.306982 7505 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40505
I20260927 22:47:49.307137 7505 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:49.307722 7506 heartbeater.cc:511] Master 127.0.97.62:41249 requested a full tablet report, sending...
I20260927 22:47:49.307758 7505 heartbeater.cc:511] Master 127.0.97.60:40505 requested a full tablet report, sending...
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 22:47:49.310206 6541 ts_manager.cc:194] Re-registered known tserver with Master: 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:49.310583 7326 ts_manager.cc:194] Registered new tserver with Master: 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:49.311228 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50489
I20260927 22:47:49.311811 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.275031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58403 (local address 127.0.97.61:39515)
0927 22:47:49.275508 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.275514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.293888 (+ 18374us) server_negotiation.cc:407] Connection header received
0927 22:47:49.293975 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.293980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.294064 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.294166 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.296901 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.298060 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.299900 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.300639 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.300739 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.305653 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.305660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.305665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.305705 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.308988 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.309760 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.309765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.309768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.309912 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.310359 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.310364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.310366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.310691 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.310830 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.311240 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.311472 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":1416,"threads_started":1}
I20260927 22:47:49.312177 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.276132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.97.1:58403)
0927 22:47:49.293630 (+ 17498us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:49.293646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:49.293749 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:49.295847 (+ 2098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:49.295851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:49.296415 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:49.296754 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.296763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.298259 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.298263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.299750 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.299763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.301036 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.301042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.301225 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.302694 (+ 1469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:49.302714 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:49.305420 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:49.309132 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.309136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.309139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.309593 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.310028 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.310033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.310037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.310193 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.310803 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:49.310807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:49.310935 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 22:47:49.311135 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 22:47:49.311509 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17391,"thread_start_us":10206,"threads_started":1}
I20260927 22:47:49.313041 7507 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39515
I20260927 22:47:49.313172 7507 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:49.313654 7507 heartbeater.cc:511] Master 127.0.97.61:39515 requested a full tablet report, sending...
I20260927 22:47:49.314975 7139 ts_manager.cc:194] Registered new tserver with Master: 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:49.318217 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.279556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.0.97.2:38613)
0927 22:47:49.294795 (+ 15239us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.294804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.294832 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:47:49.294897 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.294905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.295177 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.295347 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.298144 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.299808 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.302143 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.303269 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.304324 (+ 1055us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.312735 (+ 8411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.312746 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.312759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.312791 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.315743 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.316370 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.316377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.316379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.316490 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.316824 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.316828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.316829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.317134 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.317414 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.317685 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.317927 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15094,"thread_start_us":14947,"threads_started":1}
I20260927 22:47:49.319275 6860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:49.399417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 6737
I20260927 22:47:49.412652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38613
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41249
--webserver_port=44075
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:47:49.671129 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: true } } }
I20260927 22:47:49.671499 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:49.672708 7518 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: attempting to promote NON_VOTER 9e17792ade1a4f71bcb2d44f25e57d8e to VOTER
I20260927 22:47:49.673413 7518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } }
I20260927 22:47:49.675346 7336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 LEARNER]: Refusing update from remote peer 625e9c22217f417a926a6fa02dbf95a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:47:49.675457 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Refusing update from remote peer 625e9c22217f417a926a6fa02dbf95a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:47:49.676007 7517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:47:49.676553 7517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:47:49.679533 7518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.680256 7336 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.682466 7518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.682705 7518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:49.682965 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.683182 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:49.682960 7518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.683378 7518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:49.683930 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.687482 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.687717 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:47:49.688205 7521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "625e9c22217f417a926a6fa02dbf95a1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:49.688510 7521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: FOLLOWER
W20260927 22:47:49.722797 7516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:49.723064 7516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:49.723194 7516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:49.730050 7516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:49.730155 7516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:49.730299 7516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:49.739501 7516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44075
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:49.740722 7516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:49.741969 7516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:49.753847 7526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:49.753866 7527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:49.754001 7529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:49.754547 7516 server_base.cc:1034] running on GCE node
I20260927 22:47:49.755065 7516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:49.755966 7516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:49.757177 7516 hybrid_clock.cc:648] HybridClock initialized: now 1790549269757101 us; error 54 us; skew 500 ppm
I20260927 22:47:49.757462 7516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:49.760280 7516 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:47:49.761518 7516 webserver.cc:460] Webserver started at http://127.0.97.2:44075/ using document root <none> and password file <none>
I20260927 22:47:49.762130 7516 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:49.762255 7516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:49.767055 7516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:49.769547 7536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:49.770725 7516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:49.770859 7516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "e258531dd10243ecb9873b7f8c80091a"
format_stamp: "Formatted at 2026-09-27 22:47:46 on dist-test-slave-kr25"
server_key: "52d6da2e87e70a385949f415b798bc0d"
server_key_iv: "ff6ce4e2913d27e5976138c66955fad3"
server_key_version: "encryptionkey@0"
I20260927 22:47:49.771781 7516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:49.797478 7516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:49.801544 7516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:49.801815 7516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:49.802834 7516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:49.804242 7516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:49.804332 7516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:49.804431 7516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:49.804517 7516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:49.832935 7516 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38613
I20260927 22:47:49.833014 7648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38613 every 8 connection(s)
I20260927 22:47:49.834272 7516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:49.839174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 7516
I20260927 22:47:49.839421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 6870
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 27 22:47:49 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:47:49.867827 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.837359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53217 (local address 127.0.97.61:39515)
0927 22:47:49.837806 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.837813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.838473 (+ 660us) server_negotiation.cc:407] Connection header received
0927 22:47:49.846054 (+ 7581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.846061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.846175 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.846257 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.850853 (+ 4596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.852420 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.854732 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.855355 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.855477 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.860720 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.860727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.860731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.860782 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.863985 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.865059 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.865065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.865069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.865219 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.865768 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.865772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.865775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.866105 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.866232 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.867236 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.867485 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":446,"threads_started":1}
I20260927 22:47:49.868757 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.836993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58715 (local address 127.0.97.62:41249)
0927 22:47:49.849245 (+ 12252us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.849251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.849271 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:47:49.849330 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.849336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.849416 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.850537 (+ 1121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.850692 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.852100 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.854694 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.855338 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.855477 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.860445 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.860452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.860456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.860497 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.864547 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.865397 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.865403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.865406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.865538 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.865970 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.865975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.866004 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.866392 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.866520 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.868129 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.868375 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12124,"thread_start_us":12507,"threads_started":1}
I20260927 22:47:49.870630 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.837537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.97.2:53217)
0927 22:47:49.838320 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:49.838368 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:49.845788 (+ 7420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:49.846712 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:49.846725 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:49.847649 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:49.848465 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.848488 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.852623 (+ 4135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.852628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.854545 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.854563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.855897 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.855905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.856120 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.856847 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:49.856882 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:49.860526 (+ 3644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:49.864146 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.864156 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.864187 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.864926 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.865396 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.865401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.865404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.865650 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.866324 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:49.866331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:49.866820 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 22:47:49.868548 (+ 1728us) client_negotiation.cc:240] Negotiation successful
0927 22:47:49.868933 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":59,"spinlock_wait_cycles":38784,"thread_start_us":503,"threads_started":1}
I20260927 22:47:49.871919 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.837537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:58715)
0927 22:47:49.838665 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:49.838683 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:49.840892 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:49.849664 (+ 8772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:49.849669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:49.850121 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:49.850386 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.850397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.852290 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.852295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.854477 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.854495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.855762 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.855769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.856109 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.856993 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:49.857019 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:49.860176 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:49.864695 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.864700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.864704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.865189 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.865674 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.865678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.865681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.865841 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.866584 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:49.866588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:49.866820 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 22:47:49.867114 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 22:47:49.867669 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":339,"spinlock_wait_cycles":25600,"thread_start_us":933,"threads_started":1}
I20260927 22:47:49.872645 7651 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39515
I20260927 22:47:49.872951 7651 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:49.873111 7650 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41249
I20260927 22:47:49.873246 7650 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:49.873837 7651 heartbeater.cc:511] Master 127.0.97.61:39515 requested a full tablet report, sending...
I20260927 22:47:49.874648 7650 heartbeater.cc:511] Master 127.0.97.62:41249 requested a full tablet report, sending...
I20260927 22:47:49.875492 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.843695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39643 (local address 127.0.97.60:40505)
0927 22:47:49.856790 (+ 13095us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.856796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.857775 (+ 979us) server_negotiation.cc:407] Connection header received
0927 22:47:49.857907 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.857912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.857998 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.858092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.859337 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.860412 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.862712 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.863358 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.863508 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.867217 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.867222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.867226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.867264 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.870717 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.871741 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.871745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.871748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.871922 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.872330 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.872335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.872338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.873683 (+ 1345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.873826 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.874894 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.875094 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12977,"thread_start_us":13204,"threads_started":1}
I20260927 22:47:49.875735 6541 ts_manager.cc:194] Re-registered known tserver with Master: e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613)
I20260927 22:47:49.875845 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.843287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.97.2:39643)
0927 22:47:49.857669 (+ 14382us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:49.857687 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:49.857803 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:49.858263 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:49.858272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:49.858830 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:49.859172 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.859184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.860558 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.860563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.862518 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:49.862533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.863717 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.863724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:49.863903 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.864401 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:49.864423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:49.867015 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:49.870872 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.870877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.870881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.871611 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.872053 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:49.872058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:49.872061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:49.872210 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:49.873892 (+ 1682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:49.873897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:49.874037 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 22:47:49.874155 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:47:49.874407 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14262,"thread_start_us":14115,"threads_started":1}
I20260927 22:47:49.876616 7649 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40505
I20260927 22:47:49.876724 7649 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:49.876817 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58715
I20260927 22:47:49.877087 7649 heartbeater.cc:511] Master 127.0.97.60:40505 requested a full tablet report, sending...
I20260927 22:47:49.877990 7139 ts_manager.cc:194] Registered new tserver with Master: e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613)
I20260927 22:47:49.878211 7326 ts_manager.cc:194] Registered new tserver with Master: e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613)
I20260927 22:47:49.880898 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:49.840314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.0.97.3:41383)
0927 22:47:49.852448 (+ 12134us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:49.852456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:49.853148 (+ 692us) server_negotiation.cc:407] Connection header received
0927 22:47:49.853435 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:49.853444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:49.853682 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:49.853794 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:49.856415 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.857836 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.859692 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:49.860511 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:49.860606 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:49.871860 (+ 11254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:49.871877 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:49.871893 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:49.871934 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:49.875849 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.878506 (+ 2657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.878515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.878519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.878666 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:49.879126 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:49.879131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:49.879135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:49.879572 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:49.879840 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:49.880409 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 22:47:49.880636 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11976,"thread_start_us":12132,"threads_started":1}
I20260927 22:47:49.882654 6993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:49.968194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 6870
I20260927 22:47:49.981163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41383
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41249
--webserver_port=41129
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:50.290601 7662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:50.290860 7662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:50.290987 7662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:50.298267 7662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:50.298389 7662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:50.298544 7662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:50.308337 7662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41129
--tserver_master_addrs=127.0.97.62:41249,127.0.97.61:39515,127.0.97.60:40505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:50.309616 7662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:50.310884 7662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:47:50.313807 7506 heartbeater.cc:503] Master 127.0.97.62:41249 was elected leader, sending a full tablet report...
W20260927 22:47:50.323796 7667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:50.323819 7668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:50.323850 7670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:50.324963 7662 server_base.cc:1034] running on GCE node
I20260927 22:47:50.325525 7662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:50.326409 7662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:50.327585 7662 hybrid_clock.cc:648] HybridClock initialized: now 1790549270327533 us; error 40 us; skew 500 ppm
I20260927 22:47:50.327875 7662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:50 dist-test-slave-kr25 krb5kdc[6497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:50.330675 7662 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:47:50.332022 7662 webserver.cc:460] Webserver started at http://127.0.97.3:41129/ using document root <none> and password file <none>
I20260927 22:47:50.332623 7662 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:50.332731 7662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:50.337826 7662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:47:50.340350 7677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:50.341569 7662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:50.341708 7662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "abec420b66864c4db2a4ebb9074bd19f"
format_stamp: "Formatted at 2026-09-27 22:47:47 on dist-test-slave-kr25"
server_key: "7d84230a59b239046b1ab5aa5d5b0bb0"
server_key_iv: "bc62e12a7f942d316c76d6daac794a62"
server_key_version: "encryptionkey@0"
I20260927 22:47:50.342604 7662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:50.355033 7662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:50.358655 7662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:50.358898 7662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:50.359787 7662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:47:50.361030 7662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:50.361114 7662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:50.361209 7662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:50.361279 7662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:50.390393 7662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41383
I20260927 22:47:50.390460 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41383 every 8 connection(s)
I20260927 22:47:50.391799 7662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:50.398818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 7662
Sep 27 22:47:50 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:47:50 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:47:50 dist-test-slave-kr25 krb5kdc[6497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:50.439988 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.395844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44759 (local address 127.0.97.60:40505)
0927 22:47:50.412670 (+ 16826us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.412677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.412706 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:47:50.412765 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.412770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.412851 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.412934 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.415838 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.416710 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.418905 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.419542 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.419632 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.430812 (+ 11180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.430819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.430823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.430864 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.434916 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.436918 (+ 2002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.436924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.436929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.437078 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.437662 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.437668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.437671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.438057 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.438176 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.439320 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.439579 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16699,"thread_start_us":17173,"threads_started":1}
I20260927 22:47:50.441984 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.395910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33391 (local address 127.0.97.62:41249)
0927 22:47:50.405257 (+ 9347us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.405264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.410391 (+ 5127us) server_negotiation.cc:407] Connection header received
0927 22:47:50.411889 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.411896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.411986 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.412102 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.414497 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.415639 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.424453 (+ 8814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.425147 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.425274 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.434682 (+ 9408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.434689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.434694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.434736 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.438385 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.439523 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.439528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.439532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.439684 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.440455 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.440460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.440464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.440803 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.440957 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.441465 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.441674 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9206,"thread_start_us":9084,"threads_started":1}
I20260927 22:47:50.443104 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.397912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.3:33391)
0927 22:47:50.410281 (+ 12369us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.410302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.411756 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.412451 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.412463 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.413438 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:50.414324 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.414345 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.415786 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.415791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.424303 (+ 8512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.424319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.425508 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.425514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.425681 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.427164 (+ 1483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:50.427187 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:50.434475 (+ 7288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:50.438492 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:50.438497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:50.438501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:50.438963 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:50.439773 (+ 810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:50.439777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:50.439781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:50.439914 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:50.440936 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:50.440940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:50.441075 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.441221 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.441751 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12247,"mutex_wait_us":6401,"thread_start_us":8300,"threads_started":1}
I20260927 22:47:50.443673 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.395702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49717 (local address 127.0.97.61:39515)
0927 22:47:50.415555 (+ 19853us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.415561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.415579 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:50.415641 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.415645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.415722 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.415815 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.418716 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.419783 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.422526 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.423032 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.423150 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.430314 (+ 7164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.430321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.430325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.430366 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.440308 (+ 9942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.441109 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.441115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.441118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.441261 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.442260 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.442265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.442268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.442572 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.442713 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.443119 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.443313 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19740,"thread_start_us":20190,"threads_started":1}
I20260927 22:47:50.444169 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.396315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.97.3:49717)
0927 22:47:50.408205 (+ 11890us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.408225 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.411364 (+ 3139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.417635 (+ 6271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.417640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.418257 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:50.418575 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.418587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.420718 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.420724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.422371 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.422386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.423210 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.423216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.423553 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.424248 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:50.424276 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:50.430041 (+ 5765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:50.440454 (+ 10413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:50.440460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:50.440464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:50.440983 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:50.441354 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:50.441359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:50.441363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:50.441527 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:50.442690 (+ 1163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:50.442695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:50.442847 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.443006 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.443564 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":494,"threads_started":1}
I20260927 22:47:50.444615 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.396961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.97.3:44759)
0927 22:47:50.406693 (+ 9732us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.406753 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.411359 (+ 4606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.413720 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.413723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.414017 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:50.414319 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.414337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.416900 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.416905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.418745 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.418759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.419987 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.419995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.420304 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.421183 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:50.421214 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:50.430566 (+ 9352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:50.436093 (+ 5527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:50.436105 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:50.436128 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:50.436772 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:50.437269 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:50.437275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:50.437279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:50.437542 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:50.438334 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:50.438344 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:50.438873 (+ 529us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.441221 (+ 2348us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.441944 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9416,"mutex_wait_us":25,"thread_start_us":13441,"threads_started":1}
I20260927 22:47:50.445717 7792 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39515
I20260927 22:47:50.446059 7792 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:50.446988 7792 heartbeater.cc:511] Master 127.0.97.61:39515 requested a full tablet report, sending...
I20260927 22:47:50.447870 7790 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40505
I20260927 22:47:50.447993 7790 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:50.448381 7790 heartbeater.cc:511] Master 127.0.97.60:40505 requested a full tablet report, sending...
I20260927 22:47:50.449929 7326 ts_manager.cc:194] Registered new tserver with Master: abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383)
I20260927 22:47:50.451583 7139 ts_manager.cc:194] Registered new tserver with Master: abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383)
I20260927 22:47:50.452736 7791 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41249
I20260927 22:47:50.452847 7791 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:50.453233 7791 heartbeater.cc:511] Master 127.0.97.62:41249 requested a full tablet report, sending...
I20260927 22:47:50.454787 6541 ts_manager.cc:194] Re-registered known tserver with Master: abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383)
I20260927 22:47:50.455874 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33391
I20260927 22:47:50.465930 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.416830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.0.97.60:40505)
0927 22:47:50.434263 (+ 17433us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.434269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.447840 (+ 13571us) server_negotiation.cc:407] Connection header received
0927 22:47:50.448188 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.448194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.448282 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.448355 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.451152 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.452344 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.454256 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.455027 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.455138 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.459880 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.459887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.460669 (+ 782us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.460709 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.463259 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.464006 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.464012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.464016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.464147 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.464500 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.464506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.464509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.464841 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.465071 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.465400 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.465598 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17181,"thread_start_us":17218,"threads_started":1}
I20260927 22:47:50.469242 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.419552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.0.97.62:41249)
0927 22:47:50.442200 (+ 22648us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.442206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.453494 (+ 11288us) server_negotiation.cc:407] Connection header received
0927 22:47:50.453547 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.453552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.453636 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.453733 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.455910 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.457195 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.459948 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.460426 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.460540 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.463338 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.463344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.463348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.463381 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.466692 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.467405 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.467410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.467413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.467567 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.467901 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.467906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.467909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.468200 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.468417 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.468681 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.468879 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22548,"thread_start_us":32550,"threads_started":1}
I20260927 22:47:50.469696 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.448804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.0.97.61:39515)
0927 22:47:50.452232 (+ 3428us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.452238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.454773 (+ 2535us) server_negotiation.cc:407] Connection header received
0927 22:47:50.454987 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.454991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.455055 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.455113 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.457495 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.458500 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.460206 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.460770 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.460884 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.464075 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.464082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.464087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.464140 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.467307 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.467972 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.467976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.467978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.468088 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.468478 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.468483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.468485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.468783 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.469017 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.469207 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.469402 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3300}
I20260927 22:47:50.472739 6541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:50.480571 6541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:50.524152 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.503301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38613 (local address 127.0.0.1:42270)
0927 22:47:50.513091 (+ 9790us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.513112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.513258 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.514373 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.514378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.514404 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:50.514819 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.514833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.519687 (+ 4854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.519692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.521334 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.521345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.523263 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.523267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.523417 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.523495 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.523584 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.523723 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9648,"mutex_wait_us":4,"thread_start_us":12418,"threads_started":1}
I20260927 22:47:50.524273 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.516269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.0.97.1:38275)
0927 22:47:50.517441 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.517450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.518072 (+ 622us) server_negotiation.cc:407] Connection header received
0927 22:47:50.518208 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.518214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.518376 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.518522 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:50.518967 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.520121 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.522125 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.523276 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.523400 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.523638 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.523840 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.524003 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":936,"threads_started":1}
I20260927 22:47:50.524565 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.503698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41383 (local address 127.0.0.1:60242)
0927 22:47:50.515955 (+ 12257us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.515972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.516101 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.516591 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.516595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.516618 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:50.517009 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.517021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.518943 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.518948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.521423 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.521437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.523790 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.523794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.523918 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.523940 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.524023 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.524167 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12143,"mutex_wait_us":1,"thread_start_us":9090,"threads_started":1}
I20260927 22:47:50.524941 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.503481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.0.97.2:38613)
0927 22:47:50.513738 (+ 10257us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.513747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.513775 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:47:50.513884 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.513894 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.514143 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.515033 (+ 890us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:50.515246 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.519549 (+ 4303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.521472 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.522827 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.522953 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.523155 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.524509 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.524680 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10113,"thread_start_us":9999,"threads_started":1}
I20260927 22:47:50.525023 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.517498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38275 (local address 127.0.0.1:51358)
0927 22:47:50.517971 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.517983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.518087 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.518524 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.518528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.518548 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:50.518845 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.518853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.520263 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.520268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.521991 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.522001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.523466 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.523472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.523641 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.523668 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.524512 (+ 844us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.524641 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260927 22:47:50.525049 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.504093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.0.97.3:41383)
0927 22:47:50.504686 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.504698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.516076 (+ 11378us) server_negotiation.cc:407] Connection header received
0927 22:47:50.516247 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.516255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.516432 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.516588 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:50.517333 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.518814 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.521550 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.522864 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.522962 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.523173 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.524083 (+ 910us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.524611 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":247,"threads_started":1}
I20260927 22:47:50.528784 7725 tablet_service.cc:1467] Processing CreateTablet for tablet 2bca04dbff7b462cb73729d870c778b6 (DEFAULT_TABLE table=test-table [id=55abc6828dbb4feea1e83fa985c8bab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:50.528784 7584 tablet_service.cc:1467] Processing CreateTablet for tablet 2bca04dbff7b462cb73729d870c778b6 (DEFAULT_TABLE table=test-table [id=55abc6828dbb4feea1e83fa985c8bab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:50.529407 7440 tablet_service.cc:1467] Processing CreateTablet for tablet 2bca04dbff7b462cb73729d870c778b6 (DEFAULT_TABLE table=test-table [id=55abc6828dbb4feea1e83fa985c8bab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:50.530238 7584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bca04dbff7b462cb73729d870c778b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:50.530229 7725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bca04dbff7b462cb73729d870c778b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:50.530726 7440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bca04dbff7b462cb73729d870c778b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:50.541850 7823 tablet_bootstrap.cc:492] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Bootstrap starting.
I20260927 22:47:50.544822 7823 tablet_bootstrap.cc:654] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:50.545110 7824 tablet_bootstrap.cc:492] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Bootstrap starting.
I20260927 22:47:50.546160 7823 log.cc:824] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:50.548028 7825 tablet_bootstrap.cc:492] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Bootstrap starting.
I20260927 22:47:50.548496 7824 tablet_bootstrap.cc:654] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:50.549405 7823 tablet_bootstrap.cc:492] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: No bootstrap required, opened a new log
I20260927 22:47:50.549650 7823 ts_tablet_manager.cc:1397] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:47:50.549638 7824 log.cc:824] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:50.552052 7825 tablet_bootstrap.cc:654] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:50.553231 7825 log.cc:824] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:50.555941 7825 tablet_bootstrap.cc:492] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: No bootstrap required, opened a new log
I20260927 22:47:50.556164 7825 ts_tablet_manager.cc:1397] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:47:50.556378 7824 tablet_bootstrap.cc:492] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: No bootstrap required, opened a new log
I20260927 22:47:50.556615 7824 ts_tablet_manager.cc:1397] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:47:50.558092 7823 raft_consensus.cc:348] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.558487 7823 raft_consensus.cc:374] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:50.558573 7823 raft_consensus.cc:729] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b02ea733a314449a01b1db56ad1d2ff, State: Initialized, Role: FOLLOWER
I20260927 22:47:50.559131 7823 consensus_queue.cc:260] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.561277 7823 ts_tablet_manager.cc:1428] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:47:50.564224 7825 raft_consensus.cc:348] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.564604 7825 raft_consensus.cc:374] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:50.564700 7825 raft_consensus.cc:729] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abec420b66864c4db2a4ebb9074bd19f, State: Initialized, Role: FOLLOWER
I20260927 22:47:50.565328 7825 consensus_queue.cc:260] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.566239 7824 raft_consensus.cc:348] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.566620 7824 raft_consensus.cc:374] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:50.566720 7824 raft_consensus.cc:729] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e258531dd10243ecb9873b7f8c80091a, State: Initialized, Role: FOLLOWER
I20260927 22:47:50.567301 7824 consensus_queue.cc:260] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.568058 7825 ts_tablet_manager.cc:1428] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:47:50.568260 7650 heartbeater.cc:503] Master 127.0.97.62:41249 was elected leader, sending a full tablet report...
I20260927 22:47:50.568830 7824 ts_tablet_manager.cc:1428] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:47:50.569700 7791 heartbeater.cc:503] Master 127.0.97.62:41249 was elected leader, sending a full tablet report...
W20260927 22:47:50.587342 7652 tablet.cc:2369] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:50.606242 7830 raft_consensus.cc:482] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:50.606525 7830 raft_consensus.cc:504] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.608237 7830 leader_election.cc:288] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613), 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:50.613410 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.608546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34623 (local address 127.0.97.2:38613)
0927 22:47:50.608724 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.608729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.608747 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:50.608922 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.608927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.609001 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.609080 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:50.609423 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.610106 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.611816 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.612588 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.612699 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.612788 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.613052 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.613168 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:47:50.613443 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.608464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38613 (local address 127.0.97.3:34623)
0927 22:47:50.608612 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.608632 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.608751 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.609102 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.609106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.609130 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:50.609294 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.609301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.610225 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.610230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.611685 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.611697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.612722 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.612728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.612919 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.612941 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.613026 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.613157 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":76}
I20260927 22:47:50.613579 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.608911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59465 (local address 127.0.97.1:38275)
0927 22:47:50.609099 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.609104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.609124 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:47:50.609187 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.609192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.609252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.609322 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:50.609673 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.610374 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.611996 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.612735 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.612840 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.612925 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.613215 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.613332 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:47:50.614097 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.608796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38275 (local address 127.0.97.3:59465)
0927 22:47:50.608938 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:50.608954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:50.609076 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:50.609346 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:50.609351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:50.609372 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:50.609546 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.609553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.610517 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.610522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.611864 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:50.611875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.612872 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.612878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:50.613084 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.613107 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:47:50.613180 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:47:50.613315 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":4}
I20260927 22:47:50.614512 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bca04dbff7b462cb73729d870c778b6" candidate_uuid: "abec420b66864c4db2a4ebb9074bd19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e258531dd10243ecb9873b7f8c80091a" is_pre_election: true
I20260927 22:47:50.614910 7604 raft_consensus.cc:2454] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abec420b66864c4db2a4ebb9074bd19f in term 0.
I20260927 22:47:50.615525 7680 leader_election.cc:302] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abec420b66864c4db2a4ebb9074bd19f, e258531dd10243ecb9873b7f8c80091a; no voters:
I20260927 22:47:50.615906 7830 raft_consensus.cc:2781] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:50.616024 7830 raft_consensus.cc:482] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:50.616109 7830 raft_consensus.cc:3037] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:50.616001 7460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bca04dbff7b462cb73729d870c778b6" candidate_uuid: "abec420b66864c4db2a4ebb9074bd19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b02ea733a314449a01b1db56ad1d2ff" is_pre_election: true
I20260927 22:47:50.616443 7460 raft_consensus.cc:2454] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abec420b66864c4db2a4ebb9074bd19f in term 0.
I20260927 22:47:50.618291 7830 raft_consensus.cc:504] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.618765 7830 leader_election.cc:288] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [CANDIDATE]: Term 1 election: Requested vote from peers e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613), 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:50.619105 7604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bca04dbff7b462cb73729d870c778b6" candidate_uuid: "abec420b66864c4db2a4ebb9074bd19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e258531dd10243ecb9873b7f8c80091a"
I20260927 22:47:50.619275 7604 raft_consensus.cc:3037] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:50.619663 7460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bca04dbff7b462cb73729d870c778b6" candidate_uuid: "abec420b66864c4db2a4ebb9074bd19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b02ea733a314449a01b1db56ad1d2ff"
I20260927 22:47:50.619824 7460 raft_consensus.cc:3037] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:50.621338 7604 raft_consensus.cc:2454] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abec420b66864c4db2a4ebb9074bd19f in term 1.
I20260927 22:47:50.621649 7680 leader_election.cc:302] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abec420b66864c4db2a4ebb9074bd19f, e258531dd10243ecb9873b7f8c80091a; no voters:
I20260927 22:47:50.621901 7830 raft_consensus.cc:2781] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:50.621956 7460 raft_consensus.cc:2454] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abec420b66864c4db2a4ebb9074bd19f in term 1.
I20260927 22:47:50.622352 7830 raft_consensus.cc:686] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 1 LEADER]: Becoming Leader. State: Replica: abec420b66864c4db2a4ebb9074bd19f, State: Running, Role: LEADER
I20260927 22:47:50.622849 7830 consensus_queue.cc:237] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:50.627811 6542 catalog_manager.cc:5495] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f reported cstate change: term changed from 0 to 1, leader changed from <none> to abec420b66864c4db2a4ebb9074bd19f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "abec420b66864c4db2a4ebb9074bd19f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } health_report { overall_health: UNKNOWN } } }
W20260927 22:47:50.661597 7793 tablet.cc:2369] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:50.665683 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.660554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.0.97.3:41383)
0927 22:47:50.660722 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.660727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.660743 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:47:50.660820 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.660823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.660902 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.661019 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:50.661330 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.662376 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.663712 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.664099 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.664204 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.664649 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:50.665141 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:50.665268 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.665359 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.665437 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:47:50.674599 7604 raft_consensus.cc:1261] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 1 FOLLOWER]: Refusing update from remote peer abec420b66864c4db2a4ebb9074bd19f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:50.674619 7460 raft_consensus.cc:1261] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Refusing update from remote peer abec420b66864c4db2a4ebb9074bd19f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:50.675566 7832 consensus_queue.cc:1035] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:50.675936 7830 consensus_queue.cc:1035] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:50.686681 7835 mvcc.cc:205] Tried to move back new op lower bound from 7334089812673388544 to 7334089812475133952. Current Snapshot: MvccSnapshot[applied={T|T < 7334089812673388544}]
I20260927 22:47:50.690827 7836 mvcc.cc:205] Tried to move back new op lower bound from 7334089812673388544 to 7334089812475133952. Current Snapshot: MvccSnapshot[applied={T|T < 7334089812673388544}]
I20260927 22:47:50.693140 7837 mvcc.cc:205] Tried to move back new op lower bound from 7334089812673388544 to 7334089812475133952. Current Snapshot: MvccSnapshot[applied={T|T < 7334089812673388544}]
I20260927 22:47:50.707974 6542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:50.708212 6542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:50.714823 6542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 625e9c22217f417a926a6fa02dbf95a1: Sending DeleteTablet for 3 replicas of tablet 2bca04dbff7b462cb73729d870c778b6
I20260927 22:47:50.715878 7725 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca04dbff7b462cb73729d870c778b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:50 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60242
I20260927 22:47:50.715852 7584 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca04dbff7b462cb73729d870c778b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:50 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42270
I20260927 22:47:50.716128 7440 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca04dbff7b462cb73729d870c778b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:50 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51358
I20260927 22:47:50.725844 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 6510
I20260927 22:47:50.726773 6572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:50.727373 7846 tablet_replica.cc:330] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: stopping tablet replica
I20260927 22:47:50.728016 7846 raft_consensus.cc:2229] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:50.728609 7846 raft_consensus.cc:2258] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:50.731374 7846 ts_tablet_manager.cc:1905] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:50.734701 7847 tablet_replica.cc:330] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: stopping tablet replica
I20260927 22:47:50.735086 7847 raft_consensus.cc:2229] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:50.735589 7847 raft_consensus.cc:2258] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:50.738415 7847 ts_tablet_manager.cc:1905] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:50.738655 7846 ts_tablet_manager.cc:1918] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:50.738849 7846 log.cc:1196] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/2bca04dbff7b462cb73729d870c778b6
I20260927 22:47:50.739413 7846 ts_tablet_manager.cc:1939] T 2bca04dbff7b462cb73729d870c778b6 P e258531dd10243ecb9873b7f8c80091a: Deleting consensus metadata
I20260927 22:47:50.745345 7847 ts_tablet_manager.cc:1918] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:50.745508 7847 log.cc:1196] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/2bca04dbff7b462cb73729d870c778b6
I20260927 22:47:50.745985 7845 tablet_replica.cc:330] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: stopping tablet replica
I20260927 22:47:50.746073 7847 ts_tablet_manager.cc:1939] T 2bca04dbff7b462cb73729d870c778b6 P 4b02ea733a314449a01b1db56ad1d2ff: Deleting consensus metadata
I20260927 22:47:50.746567 7845 raft_consensus.cc:2229] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:50.747282 7845 raft_consensus.cc:2258] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:50.749598 7845 ts_tablet_manager.cc:1905] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:50.754194 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.721389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.0.97.61:39515)
0927 22:47:50.721562 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.721569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.725152 (+ 3583us) server_negotiation.cc:407] Connection header received
0927 22:47:50.726287 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.726293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.726375 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.726443 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.731053 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.732096 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.735563 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.736185 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.736279 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.747836 (+ 11557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.747843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.747847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.747888 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.751036 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.751922 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.751927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.751931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.752070 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.752458 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.752464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.752467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.752845 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.753056 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.753603 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.753847 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:47:50.756788 7845 ts_tablet_manager.cc:1918] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:50.756951 7845 log.cc:1196] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/2bca04dbff7b462cb73729d870c778b6
I20260927 22:47:50.757486 7845 ts_tablet_manager.cc:1939] T 2bca04dbff7b462cb73729d870c778b6 P abec420b66864c4db2a4ebb9074bd19f: Deleting consensus metadata
I20260927 22:47:50.772629 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.725261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.0.97.60:40505)
0927 22:47:50.725461 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.725467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.756679 (+ 31212us) server_negotiation.cc:407] Connection header received
0927 22:47:50.756940 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.756947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.757035 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.757108 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.759205 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.760322 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.762250 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.762888 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.763016 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.766444 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.766451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.766456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.766495 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.769823 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.770593 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.770599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.770602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.770746 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.771123 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.771127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.771131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.771528 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.771749 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.772078 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.772295 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:47:50.869220 6528 catalog_manager.cc:4841] TS 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275): tablet 2bca04dbff7b462cb73729d870c778b6 (table test-table [id=55abc6828dbb4feea1e83fa985c8bab6]) successfully deleted
I20260927 22:47:50.869555 6528 catalog_manager.cc:4841] TS e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613): tablet 2bca04dbff7b462cb73729d870c778b6 (table test-table [id=55abc6828dbb4feea1e83fa985c8bab6]) successfully deleted
I20260927 22:47:50.869880 6527 catalog_manager.cc:4841] TS abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383): tablet 2bca04dbff7b462cb73729d870c778b6 (table test-table [id=55abc6828dbb4feea1e83fa985c8bab6]) successfully deleted
I20260927 22:47:50.870009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 6510
I20260927 22:47:50.920817 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.900021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.0.97.60:40505)
0927 22:47:50.900191 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.900197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.901581 (+ 1384us) server_negotiation.cc:407] Connection header received
0927 22:47:50.901769 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.901774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.901869 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.901956 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.904122 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.905076 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.906632 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.907297 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.907416 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.915249 (+ 7833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.915256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.915260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.915300 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.917648 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.918813 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.918818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.918822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.918959 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.919331 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.919336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.919339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.919691 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.919880 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.920385 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.920566 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:47:50.921022 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:50.899985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.0.97.61:39515)
0927 22:47:50.900196 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:50.900202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:50.901370 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 22:47:50.901636 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:50.901643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:50.901730 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:50.901833 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:50.903687 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.904705 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.906800 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:50.907408 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:50.907544 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:50.914566 (+ 7022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:50.914571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:50.914574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:50.914610 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:50.917735 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.918517 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.918523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.918527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.918671 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:50.919018 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:50.919023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:50.919027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:50.919420 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:50.919679 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:50.920225 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 22:47:50.920613 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:51.693476 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:51.692818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:46877)
0927 22:47:51.693196 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:51.693314 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":194,"threads_started":1}
W20260927 22:47:51.693909 7650 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41249 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:51.706631 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:51.704944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.1:41165)
0927 22:47:51.706283 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:51.706432 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1208,"threads_started":1}
W20260927 22:47:51.707367 7506 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41249 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:51.707401 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:51.704936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.3:50673)
0927 22:47:51.707067 (+ 2131us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:51.707185 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2035,"thread_start_us":1903,"threads_started":1}
W20260927 22:47:51.707769 7791 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41249 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:52.214915 7859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 625e9c22217f417a926a6fa02dbf95a1)
I20260927 22:47:52.215179 7859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } }
I20260927 22:47:52.215595 7860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 625e9c22217f417a926a6fa02dbf95a1)
I20260927 22:47:52.215819 7860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } }
I20260927 22:47:52.216602 7859 leader_election.cc:288] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 625e9c22217f417a926a6fa02dbf95a1 (127.0.97.62:41249), 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60:40505)
I20260927 22:47:52.223165 7860 leader_election.cc:288] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 625e9c22217f417a926a6fa02dbf95a1 (127.0.97.62:41249), 1b9ef0b13f3247bb942e425956a1518d (127.0.97.61:39515)
I20260927 22:47:52.224849 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.216827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56104)
0927 22:47:52.224353 (+ 7526us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.224558 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7402,"mutex_wait_us":28,"thread_start_us":6518,"threads_started":1}
W20260927 22:47:52.225247 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:52.228917 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.217176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56118)
0927 22:47:52.228501 (+ 11325us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.228660 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11216,"thread_start_us":11088,"threads_started":1}
W20260927 22:47:52.229560 7312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:52.233242 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.232728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56120)
0927 22:47:52.232917 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.233058 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:47:52.233482 7312 leader_election.cc:334] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 625e9c22217f417a926a6fa02dbf95a1 (127.0.97.62:41249): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:52.233597 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.233021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56136)
0927 22:47:52.233217 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.233414 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 22:47:52.233886 7125 leader_election.cc:334] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 625e9c22217f417a926a6fa02dbf95a1 (127.0.97.62:41249): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:52.235400 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.216827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40505 (local address 127.0.0.1:49156)
0927 22:47:52.223664 (+ 6837us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.223690 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:52.223859 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:52.228841 (+ 4982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:52.228847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:52.228895 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:52.229225 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.229239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.230891 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.230896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.233252 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.233267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.234665 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.234672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.234835 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.234864 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:47:52.234948 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:47:52.235071 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6376,"mutex_wait_us":256,"thread_start_us":7262,"threads_started":1}
I20260927 22:47:52.236133 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.217088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.97.60:40505)
0927 22:47:52.228460 (+ 11372us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.228465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.228489 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:47:52.228562 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.228567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.228653 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.229374 (+ 721us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:52.229577 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.230722 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.233443 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.234511 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.234633 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.234726 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.235698 (+ 972us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.235824 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":7524,"thread_start_us":7351,"threads_started":1}
I20260927 22:47:52.236898 7336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b9ef0b13f3247bb942e425956a1518d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" is_pre_election: true
I20260927 22:47:52.237262 7336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9ef0b13f3247bb942e425956a1518d in term 1.
I20260927 22:47:52.237753 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.224859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39515 (local address 127.0.0.1:35654)
0927 22:47:52.229130 (+ 4271us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.229147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:52.229289 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:52.230528 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:52.230533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:52.230556 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:52.230850 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.230860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.233613 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.233618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.235317 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.235329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.237003 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.237009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.237207 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.237236 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:47:52.237326 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:47:52.237453 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4177,"mutex_wait_us":1,"thread_start_us":6817,"threads_started":1}
I20260927 22:47:52.237826 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.224997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.0.97.61:39515)
0927 22:47:52.228361 (+ 3364us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.228368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.229214 (+ 846us) server_negotiation.cc:407] Connection header received
0927 22:47:52.230058 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.230063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.230209 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.230307 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:52.231031 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.233430 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.235516 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.236735 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.236850 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.236964 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.237378 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.237504 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3225,"thread_start_us":3003,"threads_started":1}
I20260927 22:47:52.237744 7123 leader_election.cc:302] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b9ef0b13f3247bb942e425956a1518d, 9e17792ade1a4f71bcb2d44f25e57d8e; no voters: 625e9c22217f417a926a6fa02dbf95a1
I20260927 22:47:52.238168 7859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:47:52.238293 7859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 625e9c22217f417a926a6fa02dbf95a1)
I20260927 22:47:52.238391 7859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:47:52.238436 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1b9ef0b13f3247bb942e425956a1518d" is_pre_election: true
I20260927 22:47:52.240507 7859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } }
I20260927 22:47:52.240875 7148 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e17792ade1a4f71bcb2d44f25e57d8e in current term 2: Already voted for candidate 1b9ef0b13f3247bb942e425956a1518d in this term.
I20260927 22:47:52.241155 7859 leader_election.cc:288] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [CANDIDATE]: Term 2 election: Requested vote from peers 625e9c22217f417a926a6fa02dbf95a1 (127.0.97.62:41249), 9e17792ade1a4f71bcb2d44f25e57d8e (127.0.97.60:40505)
I20260927 22:47:52.241549 7336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b9ef0b13f3247bb942e425956a1518d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e"
I20260927 22:47:52.241670 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.241212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56140)
0927 22:47:52.241439 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.241542 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:47:52.241724 7336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:47:52.242260 7313 leader_election.cc:302] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e17792ade1a4f71bcb2d44f25e57d8e; no voters: 1b9ef0b13f3247bb942e425956a1518d, 625e9c22217f417a926a6fa02dbf95a1
I20260927 22:47:52.243949 7336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9ef0b13f3247bb942e425956a1518d in term 2.
I20260927 22:47:52.244102 7860 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:47:52.244313 7123 leader_election.cc:302] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9ef0b13f3247bb942e425956a1518d, 9e17792ade1a4f71bcb2d44f25e57d8e; no voters:
I20260927 22:47:52.244632 7859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:47:52.245044 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.244704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56144)
0927 22:47:52.244836 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.244904 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:47:52.245088 7859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [term 2 LEADER]: Becoming Leader. State: Replica: 1b9ef0b13f3247bb942e425956a1518d, State: Running, Role: LEADER
W20260927 22:47:52.245301 7125 leader_election.cc:334] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 625e9c22217f417a926a6fa02dbf95a1 (127.0.97.62:41249): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
I20260927 22:47:52.245486 7859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } }
I20260927 22:47:52.247416 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b9ef0b13f3247bb942e425956a1518d. Latest consensus state: current_term: 2 leader_uuid: "1b9ef0b13f3247bb942e425956a1518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:52.247799 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:52.248452 7871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:52.251820 7871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:52.252609 7871 catalog_manager.cc:1259] Loaded cluster ID: 6cc4d210a841477cbc17b60811b2e259
I20260927 22:47:52.252682 7871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:52.253801 7871 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:52.254585 7871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Loaded TSK: 0
I20260927 22:47:52.255352 7871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:47:52.293107 7138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:52.300081 7138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:52.304028 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.303665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56162)
0927 22:47:52.303806 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.303892 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:47:52.304152 7336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [term 2 FOLLOWER]: Refusing update from remote peer 1b9ef0b13f3247bb942e425956a1518d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 22:47:52.304893 7869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 22:47:52.305055 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.304635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56172)
0927 22:47:52.304826 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.304917 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 22:47:52.305217 7125 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d -> Peer 625e9c22217f417a926a6fa02dbf95a1 (127.0.97.62:41249): Couldn't send request to peer 625e9c22217f417a926a6fa02dbf95a1. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:47:52.307467 7860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b9ef0b13f3247bb942e425956a1518d. Latest consensus state: current_term: 2 leader_uuid: "1b9ef0b13f3247bb942e425956a1518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:52.307745 7860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:47:52.308861 7872 mvcc.cc:205] Tried to move back new op lower bound from 7334089819351187456 to 7334089819121049600. Current Snapshot: MvccSnapshot[applied={T|T < 7334089819351187456}]
I20260927 22:47:52.309029 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1b9ef0b13f3247bb942e425956a1518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:52.309312 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:52.310299 7860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1b9ef0b13f3247bb942e425956a1518d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "625e9c22217f417a926a6fa02dbf95a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41249 } } peers { permanent_uuid: "1b9ef0b13f3247bb942e425956a1518d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39515 } attrs { promote: false } } peers { permanent_uuid: "9e17792ade1a4f71bcb2d44f25e57d8e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40505 } attrs { promote: false } } }
I20260927 22:47:52.310530 7860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17792ade1a4f71bcb2d44f25e57d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:47:52.324640 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.317098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41383 (local address 127.0.0.1:60248)
0927 22:47:52.317243 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.317267 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:52.317347 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:52.318108 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:52.318113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:52.318138 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:52.318423 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.318435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.320064 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.320070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.322255 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.322271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.323912 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.323918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.324068 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.324098 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:47:52.324191 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 22:47:52.324345 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260927 22:47:52.325346 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.317394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.0.97.3:41383)
0927 22:47:52.317787 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.317794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.317812 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:52.317881 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.317886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.317955 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.318556 (+ 601us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:52.318759 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.319930 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.322441 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.323730 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.323838 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.323939 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.324944 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.325077 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":937,"threads_started":1}
I20260927 22:47:52.325963 7725 tablet_service.cc:1467] Processing CreateTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f (DEFAULT_TABLE table=test-table [id=e524415634ed4ca69bb8924a56e33a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:52.326329 7725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bcb08b0dd045be9f8a549c8dfb154f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:52.330643 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.317705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38275 (local address 127.0.0.1:51366)
0927 22:47:52.318277 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.318294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:52.318430 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:52.324137 (+ 5707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:52.324142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:52.324168 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:52.324499 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.324509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.325885 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.325889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.327998 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.328013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.329822 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.329828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.330010 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.330045 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:47:52.330132 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 22:47:52.330267 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":642,"threads_started":1}
I20260927 22:47:52.330948 7882 tablet_bootstrap.cc:492] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: Bootstrap starting.
I20260927 22:47:52.331343 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.318070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.97.1:38275)
0927 22:47:52.323827 (+ 5757us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.323834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.323855 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:47:52.323926 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.323932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.324007 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.324115 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:52.324635 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.325726 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.328218 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.329458 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.329575 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.329681 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.330870 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.331052 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5619,"thread_start_us":5523,"threads_started":1}
I20260927 22:47:52.332062 7440 tablet_service.cc:1467] Processing CreateTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f (DEFAULT_TABLE table=test-table [id=e524415634ed4ca69bb8924a56e33a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:52.332485 7440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bcb08b0dd045be9f8a549c8dfb154f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:52.333065 7882 tablet_bootstrap.cc:654] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:52.334940 7882 tablet_bootstrap.cc:492] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: No bootstrap required, opened a new log
I20260927 22:47:52.335068 7882 ts_tablet_manager.cc:1397] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:47:52.335763 7882 raft_consensus.cc:348] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.335937 7882 raft_consensus.cc:374] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:52.336021 7882 raft_consensus.cc:729] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abec420b66864c4db2a4ebb9074bd19f, State: Initialized, Role: FOLLOWER
I20260927 22:47:52.336252 7882 consensus_queue.cc:260] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.337759 7882 ts_tablet_manager.cc:1428] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:52.338148 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.316699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38613 (local address 127.0.0.1:42274)
0927 22:47:52.316847 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.316867 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:52.316977 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:52.332600 (+ 15623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:52.332605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:52.332633 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:52.332825 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.332836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.334168 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.334173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.335913 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.335926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.337460 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.337467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.337646 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.337681 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:47:52.337750 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:47:52.337877 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 22:47:52.340426 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.316794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.0.97.2:38613)
0927 22:47:52.332289 (+ 15495us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.332295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.332313 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:52.332378 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.332383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.332453 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.332565 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:52.332997 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.334003 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.336105 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.337289 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.339805 (+ 2516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.339932 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.339985 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.340135 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15377,"thread_start_us":15874,"threads_started":1}
I20260927 22:47:52.341570 7584 tablet_service.cc:1467] Processing CreateTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f (DEFAULT_TABLE table=test-table [id=e524415634ed4ca69bb8924a56e33a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:52.342022 7584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bcb08b0dd045be9f8a549c8dfb154f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:52.346046 7886 tablet_bootstrap.cc:492] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: Bootstrap starting.
I20260927 22:47:52.347544 7886 tablet_bootstrap.cc:654] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:52.347836 7792 heartbeater.cc:503] Master 127.0.97.61:39515 was elected leader, sending a full tablet report...
I20260927 22:47:52.351250 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.338248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.3:45763)
0927 22:47:52.350803 (+ 12555us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.350948 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12420,"thread_start_us":12294,"threads_started":1}
I20260927 22:47:52.352012 7888 tablet_bootstrap.cc:492] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: Bootstrap starting.
I20260927 22:47:52.352455 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.351999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.3:49163)
0927 22:47:52.352214 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.352298 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 22:47:52.353514 7888 tablet_bootstrap.cc:654] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:52.355376 7886 tablet_bootstrap.cc:492] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: No bootstrap required, opened a new log
I20260927 22:47:52.355548 7886 ts_tablet_manager.cc:1397] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260927 22:47:52.355818 7888 tablet_bootstrap.cc:492] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: No bootstrap required, opened a new log
I20260927 22:47:52.355947 7888 ts_tablet_manager.cc:1397] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:47:52.356205 7886 raft_consensus.cc:348] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.356377 7886 raft_consensus.cc:374] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:52.356464 7886 raft_consensus.cc:729] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b02ea733a314449a01b1db56ad1d2ff, State: Initialized, Role: FOLLOWER
I20260927 22:47:52.356518 7888 raft_consensus.cc:348] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.356727 7888 raft_consensus.cc:374] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:52.356813 7888 raft_consensus.cc:729] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e258531dd10243ecb9873b7f8c80091a, State: Initialized, Role: FOLLOWER
I20260927 22:47:52.356688 7886 consensus_queue.cc:260] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.357025 7888 consensus_queue.cc:260] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.357410 7886 ts_tablet_manager.cc:1428] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:47:52.358206 7888 ts_tablet_manager.cc:1428] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:47:52.363183 7651 heartbeater.cc:503] Master 127.0.97.61:39515 was elected leader, sending a full tablet report...
I20260927 22:47:52.364267 7507 heartbeater.cc:503] Master 127.0.97.61:39515 was elected leader, sending a full tablet report...
I20260927 22:47:52.365269 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.358008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:59841)
0927 22:47:52.364925 (+ 6917us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.365074 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6772,"thread_start_us":6637,"threads_started":1}
I20260927 22:47:52.366041 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.357891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.1:60231)
0927 22:47:52.365669 (+ 7778us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.365789 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7685,"thread_start_us":7856,"threads_started":1}
I20260927 22:47:52.366178 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.365835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:52779)
0927 22:47:52.365990 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.366067 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:47:52.366873 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.366557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.1:58311)
0927 22:47:52.366712 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.366777 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:47:52.413003 7891 raft_consensus.cc:482] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:52.413201 7891 raft_consensus.cc:504] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.414706 7891 leader_election.cc:288] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383), 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:52.422199 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.415035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37231 (local address 127.0.97.3:41383)
0927 22:47:52.415217 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.415224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.415411 (+ 187us) server_negotiation.cc:407] Connection header received
0927 22:47:52.415656 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.415660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.415727 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.415825 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:52.416301 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.417321 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.419905 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.421080 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.421206 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.421314 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.421821 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.421965 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:47:52.422194 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.415032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41963 (local address 127.0.97.1:38275)
0927 22:47:52.415232 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.415238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.415741 (+ 503us) server_negotiation.cc:407] Connection header received
0927 22:47:52.415846 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.415851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.415915 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.415996 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:52.416486 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.417484 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.419899 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.421092 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.421206 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.421313 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.421805 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.421938 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:47:52.422379 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.415033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38275 (local address 127.0.97.2:41963)
0927 22:47:52.415649 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.415663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:52.415753 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:52.416023 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:52.416028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:52.416052 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:52.416368 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.416375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.417607 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.417611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.419758 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.419767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.421356 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.421362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.421583 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.421664 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 22:47:52.421762 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:47:52.421898 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":105,"thread_start_us":343,"threads_started":1}
I20260927 22:47:52.422744 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.414966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41383 (local address 127.0.97.2:37231)
0927 22:47:52.415309 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.415330 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:52.415513 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:52.415854 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:52.415857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:52.415875 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:52.416153 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.416165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.417457 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.417461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.419752 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:52.419767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.421320 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.421327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:52.421578 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.421711 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 22:47:52.421780 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:47:52.421898 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":81}
I20260927 22:47:52.423159 7460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bcb08b0dd045be9f8a549c8dfb154f" candidate_uuid: "e258531dd10243ecb9873b7f8c80091a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b02ea733a314449a01b1db56ad1d2ff" is_pre_election: true
I20260927 22:47:52.423405 7460 raft_consensus.cc:2454] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e258531dd10243ecb9873b7f8c80091a in term 0.
I20260927 22:47:52.423513 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bcb08b0dd045be9f8a549c8dfb154f" candidate_uuid: "e258531dd10243ecb9873b7f8c80091a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abec420b66864c4db2a4ebb9074bd19f" is_pre_election: true
I20260927 22:47:52.423892 7745 raft_consensus.cc:2454] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e258531dd10243ecb9873b7f8c80091a in term 0.
I20260927 22:47:52.423920 7539 leader_election.cc:302] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b02ea733a314449a01b1db56ad1d2ff, e258531dd10243ecb9873b7f8c80091a; no voters:
I20260927 22:47:52.424275 7891 raft_consensus.cc:2781] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:52.424374 7891 raft_consensus.cc:482] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:52.424466 7891 raft_consensus.cc:3037] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:52.426514 7891 raft_consensus.cc:504] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.427066 7891 leader_election.cc:288] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [CANDIDATE]: Term 1 election: Requested vote from peers abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383), 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275)
I20260927 22:47:52.427563 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bcb08b0dd045be9f8a549c8dfb154f" candidate_uuid: "e258531dd10243ecb9873b7f8c80091a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abec420b66864c4db2a4ebb9074bd19f"
I20260927 22:47:52.427563 7460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bcb08b0dd045be9f8a549c8dfb154f" candidate_uuid: "e258531dd10243ecb9873b7f8c80091a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b02ea733a314449a01b1db56ad1d2ff"
I20260927 22:47:52.427729 7745 raft_consensus.cc:3037] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:52.427737 7460 raft_consensus.cc:3037] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:52.429623 7460 raft_consensus.cc:2454] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e258531dd10243ecb9873b7f8c80091a in term 1.
I20260927 22:47:52.429679 7745 raft_consensus.cc:2454] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e258531dd10243ecb9873b7f8c80091a in term 1.
I20260927 22:47:52.430069 7539 leader_election.cc:302] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b02ea733a314449a01b1db56ad1d2ff, abec420b66864c4db2a4ebb9074bd19f, e258531dd10243ecb9873b7f8c80091a; no voters:
I20260927 22:47:52.430363 7891 raft_consensus.cc:2781] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:52.430775 7891 raft_consensus.cc:686] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 1 LEADER]: Becoming Leader. State: Replica: e258531dd10243ecb9873b7f8c80091a, State: Running, Role: LEADER
I20260927 22:47:52.431180 7891 consensus_queue.cc:237] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } }
I20260927 22:47:52.431725 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.431327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:46419)
0927 22:47:52.431498 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.431602 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:47:52.434742 7139 catalog_manager.cc:5495] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a reported cstate change: term changed from 0 to 1, leader changed from <none> to e258531dd10243ecb9873b7f8c80091a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e258531dd10243ecb9873b7f8c80091a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e258531dd10243ecb9873b7f8c80091a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:52.457289 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.452887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.0.97.2:38613)
0927 22:47:52.453058 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.453062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.453078 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:47:52.453185 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.453189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.453269 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.453387 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:52.453648 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.454326 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.455529 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.455933 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.456041 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.456359 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:52.456884 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:52.456985 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.457025 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.457083 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:47:52.461117 7460 raft_consensus.cc:1261] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Refusing update from remote peer e258531dd10243ecb9873b7f8c80091a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:52.461457 7745 raft_consensus.cc:1261] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 1 FOLLOWER]: Refusing update from remote peer e258531dd10243ecb9873b7f8c80091a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:52.461790 7895 consensus_queue.cc:1035] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b02ea733a314449a01b1db56ad1d2ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:52.462178 7891 consensus_queue.cc:1035] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [LEADER]: Connected to new peer: Peer: permanent_uuid: "abec420b66864c4db2a4ebb9074bd19f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:52.464475 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.463901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.1:58947)
0927 22:47:52.464144 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.464256 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 22:47:52.464766 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.464280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.3:50879)
0927 22:47:52.464500 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.464597 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:47:52.470079 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.469653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.1:51243)
0927 22:47:52.469855 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.469937 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 22:47:52.471490 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.470992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:45579)
0927 22:47:52.471229 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.471317 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 22:47:52.484012 7139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:52.484261 7139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:52.484719 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.484237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.2:32901)
0927 22:47:52.484451 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.484549 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 22:47:52.488662 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.487252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.97.3:33607)
0927 22:47:52.488367 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.488464 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":995}
I20260927 22:47:52.494127 7139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b9ef0b13f3247bb942e425956a1518d: Sending DeleteTablet for 3 replicas of tablet 24bcb08b0dd045be9f8a549c8dfb154f
I20260927 22:47:52.495537 7440 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:52 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51366
I20260927 22:47:52.495555 7584 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:52 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42274
I20260927 22:47:52.495647 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:52 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42274
I20260927 22:47:52.496312 7725 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:52 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60248
I20260927 22:47:52.496330 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bcb08b0dd045be9f8a549c8dfb154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:52 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60248
I20260927 22:47:52.497961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 7299
I20260927 22:47:52.508375 7909 tablet_replica.cc:330] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: stopping tablet replica
I20260927 22:47:52.508859 7909 raft_consensus.cc:2229] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:52.509186 7909 raft_consensus.cc:2258] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:52.510191 7909 ts_tablet_manager.cc:1905] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:52.515793 7908 tablet_replica.cc:330] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: stopping tablet replica
I20260927 22:47:52.515977 7908 raft_consensus.cc:2229] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:52.516458 7908 raft_consensus.cc:2258] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:52.517668 7909 ts_tablet_manager.cc:1918] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:52.517772 7909 log.cc:1196] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/24bcb08b0dd045be9f8a549c8dfb154f
I20260927 22:47:52.518108 7909 ts_tablet_manager.cc:1939] T 24bcb08b0dd045be9f8a549c8dfb154f P abec420b66864c4db2a4ebb9074bd19f: Deleting consensus metadata
I20260927 22:47:52.519109 7124 catalog_manager.cc:4841] TS abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383): tablet 24bcb08b0dd045be9f8a549c8dfb154f (table test-table [id=e524415634ed4ca69bb8924a56e33a8e]) successfully deleted
W20260927 22:47:52.519598 7124 catalog_manager.cc:4808] TS abec420b66864c4db2a4ebb9074bd19f (127.0.97.3:41383): delete failed for tablet 24bcb08b0dd045be9f8a549c8dfb154f because the tablet was not found. No further retry: Not found: Tablet not found: 24bcb08b0dd045be9f8a549c8dfb154f
I20260927 22:47:52.519995 7907 tablet_replica.cc:330] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: stopping tablet replica
I20260927 22:47:52.520083 7908 ts_tablet_manager.cc:1905] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:52.523614 7907 raft_consensus.cc:2229] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:52.524001 7907 raft_consensus.cc:2258] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:52.526026 7907 ts_tablet_manager.cc:1905] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:52.526897 7908 ts_tablet_manager.cc:1918] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:52.527055 7908 log.cc:1196] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/24bcb08b0dd045be9f8a549c8dfb154f
I20260927 22:47:52.527573 7908 ts_tablet_manager.cc:1939] T 24bcb08b0dd045be9f8a549c8dfb154f P e258531dd10243ecb9873b7f8c80091a: Deleting consensus metadata
I20260927 22:47:52.529523 7125 catalog_manager.cc:4841] TS e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613): tablet 24bcb08b0dd045be9f8a549c8dfb154f (table test-table [id=e524415634ed4ca69bb8924a56e33a8e]) successfully deleted
W20260927 22:47:52.529843 7125 catalog_manager.cc:4808] TS e258531dd10243ecb9873b7f8c80091a (127.0.97.2:38613): delete failed for tablet 24bcb08b0dd045be9f8a549c8dfb154f because the tablet was not found. No further retry: Not found: Tablet not found: 24bcb08b0dd045be9f8a549c8dfb154f
I20260927 22:47:52.532912 7907 ts_tablet_manager.cc:1918] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:52.533059 7907 log.cc:1196] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/24bcb08b0dd045be9f8a549c8dfb154f
I20260927 22:47:52.533521 7907 ts_tablet_manager.cc:1939] T 24bcb08b0dd045be9f8a549c8dfb154f P 4b02ea733a314449a01b1db56ad1d2ff: Deleting consensus metadata
I20260927 22:47:52.534190 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.499088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.0.97.1:38275)
0927 22:47:52.499274 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.499281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.518680 (+ 19399us) server_negotiation.cc:407] Connection header received
0927 22:47:52.519874 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.519880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.519961 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.520059 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:52.521840 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.522949 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.524585 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.525192 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.525324 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.528413 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:52.528428 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:52.528448 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:52.528506 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:52.531687 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:52.532318 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:52.532325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:52.532327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:52.532442 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:52.532781 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:52.532785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:52.532787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:52.533182 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:52.533513 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.533752 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.533933 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:47:52.534622 7125 catalog_manager.cc:4841] TS 4b02ea733a314449a01b1db56ad1d2ff (127.0.97.1:38275): tablet 24bcb08b0dd045be9f8a549c8dfb154f (table test-table [id=e524415634ed4ca69bb8924a56e33a8e]) successfully deleted
I20260927 22:47:52.535292 7500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:52.674786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 7299
I20260927 22:47:52.688823 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 7516
I20260927 22:47:52.701992 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.689516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.0.97.2:38613)
0927 22:47:52.689993 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.689997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.690010 (+ 13us) server_negotiation.cc:407] Connection header received
0927 22:47:52.690057 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.690061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.690118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.690197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:52.691401 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.692215 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.693410 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.693847 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.693961 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.696328 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:52.696340 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:52.696358 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:52.696390 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:52.699456 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:52.700078 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:52.700087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:52.700090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:52.700250 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:52.700592 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:52.700597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:52.700599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:52.700934 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:52.701239 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.701475 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.701686 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260927 22:47:52.702689 7644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:52.807960 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.807569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56186)
0927 22:47:52.807726 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.807820 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 22:47:52.809165 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.808857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41249 (local address 127.0.0.1:56188)
0927 22:47:52.808994 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:52.809061 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 22:47:52.815955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 7516
I20260927 22:47:52.831472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 7662
I20260927 22:47:52.843807 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:52.832148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.0.97.3:41383)
0927 22:47:52.832317 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:52.832321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:52.832355 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:47:52.832508 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:52.832511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:52.832569 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:52.832653 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:52.833851 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.834525 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.835718 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:52.836121 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:52.836209 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:52.838565 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:52.838578 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:52.838594 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:52.838623 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:52.841266 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:52.841936 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:52.841944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:52.841946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:52.842097 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:52.842431 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:52.842434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:52.842436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:52.842786 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:52.843022 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:52.843210 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 22:47:52.843368 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:47:52.844434 7785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:52.957192 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 7662
I20260927 22:47:52.973799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 7003
I20260927 22:47:52.974759 7169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:53.093914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 7003
I20260927 22:47:53.110953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 7103
I20260927 22:47:53.111974 7356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:53.221383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 7103
2026-09-27T22:47:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8875 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:53 dist-test-slave-kr25 krb5kdc[7919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:53 dist-test-slave-kr25 krb5kdc[7919](info): set up 2 sockets
Sep 27 22:47:53 dist-test-slave-kr25 krb5kdc[7919](info): commencing operation
krb5kdc: starting...
W20260927 22:47:54.240468 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:54 dist-test-slave-kr25 krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 22:47:54.433288 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43877
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:54.738343 7932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:54.738624 7932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:54.738762 7932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:54.745748 7932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:54.745857 7932 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:47:54.745924 7932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:54.746011 7932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:54.746078 7932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:54.746157 7932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:54.755760 7932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43877
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:54.756968 7932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:54.758211 7932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:54.768699 7937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:54.769009 7940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:54.769105 7938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:54.769999 7932 server_base.cc:1034] running on GCE node
I20260927 22:47:54.770699 7932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:54.772071 7932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:54.773293 7932 hybrid_clock.cc:648] HybridClock initialized: now 1790549274773228 us; error 38 us; skew 500 ppm
I20260927 22:47:54.773612 7932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:54 dist-test-slave-kr25 krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:54.776609 7932 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 22:47:54.777935 7932 webserver.cc:460] Webserver started at http://127.0.97.62:34119/ using document root <none> and password file <none>
I20260927 22:47:54.778600 7932 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:54.778704 7932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:54.779019 7932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:54.781935 7932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "7add240533804303979ff30267a2699d"
format_stamp: "Formatted at 2026-09-27 22:47:54 on dist-test-slave-kr25"
server_key: "2c5c0c8e89959e430ad5e56772ef728b"
server_key_iv: "070d91bc77cea0284cfe66df1ac84b02"
server_key_version: "encryptionkey@0"
I20260927 22:47:54.782738 7932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "7add240533804303979ff30267a2699d"
format_stamp: "Formatted at 2026-09-27 22:47:54 on dist-test-slave-kr25"
server_key: "2c5c0c8e89959e430ad5e56772ef728b"
server_key_iv: "070d91bc77cea0284cfe66df1ac84b02"
server_key_version: "encryptionkey@0"
I20260927 22:47:54.787823 7932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:47:54.791141 7947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:54.792438 7932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:54.792604 7932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "7add240533804303979ff30267a2699d"
format_stamp: "Formatted at 2026-09-27 22:47:54 on dist-test-slave-kr25"
server_key: "2c5c0c8e89959e430ad5e56772ef728b"
server_key_iv: "070d91bc77cea0284cfe66df1ac84b02"
server_key_version: "encryptionkey@0"
I20260927 22:47:54.792805 7932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:54.812108 7932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:54.815828 7932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:54.816046 7932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:54.833948 7932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43877
I20260927 22:47:54.833967 8008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43877 every 8 connection(s)
I20260927 22:47:54.835158 7932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:47:54.838634 8009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:54.841320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 7932
I20260927 22:47:54.841492 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:47:54.842080 388 external_mini_cluster.cc:1428] Setting key 067626a4a3bfb46920ffcf4d58c558a1
I20260927 22:47:54.850553 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Bootstrap starting.
I20260927 22:47:54.853859 8009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:54.854748 8009 log.cc:824] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:47:54 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:54.857467 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: No bootstrap required, opened a new log
I20260927 22:47:54.863485 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:54.843882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.97.62:43877)
0927 22:47:54.844614 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:54.844628 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:54.844675 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:47:54.846611 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:54.846642 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:54.847143 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:54.847578 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:54.848752 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:54.850599 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:54.851880 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:54.852869 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:54.853050 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:54.856812 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:54.856819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:54.856833 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:54.856869 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:54.859664 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:54.860401 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:54.860410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:54.860415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:54.860559 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:54.860927 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:54.860930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:54.860935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:54.861409 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:54.861645 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:54.862082 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 22:47:54.862455 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":289,"threads_started":1}
I20260927 22:47:54.865778 8009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add240533804303979ff30267a2699d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43877 } }
I20260927 22:47:54.866189 8009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:54.866288 8009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7add240533804303979ff30267a2699d, State: Initialized, Role: FOLLOWER
I20260927 22:47:54.866822 8009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add240533804303979ff30267a2699d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43877 } }
I20260927 22:47:54.867034 8009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:47:54.867151 8009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:47:54.867288 8009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:54.869305 8009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add240533804303979ff30267a2699d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43877 } }
I20260927 22:47:54.869766 8009 leader_election.cc:302] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7add240533804303979ff30267a2699d; no voters:
I20260927 22:47:54.870296 8009 leader_election.cc:288] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:47:54.870398 8014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:54.871416 8014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [term 1 LEADER]: Becoming Leader. State: Replica: 7add240533804303979ff30267a2699d, State: Running, Role: LEADER
I20260927 22:47:54.871944 8014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add240533804303979ff30267a2699d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43877 } }
I20260927 22:47:54.872411 8009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:47:54.874366 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7add240533804303979ff30267a2699d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add240533804303979ff30267a2699d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43877 } } }
I20260927 22:47:54.874693 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:54.875595 8021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:47:54.875417 8016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7add240533804303979ff30267a2699d. Latest consensus state: current_term: 1 leader_uuid: "7add240533804303979ff30267a2699d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7add240533804303979ff30267a2699d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43877 } } }
I20260927 22:47:54.878225 8016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d [sys.catalog]: This master's current role is: LEADER
I20260927 22:47:54.880776 8021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:47:54.888615 8021 catalog_manager.cc:1347] Generated new cluster ID: 170ad8e0199f4bb8b61e67b64d4f3757
I20260927 22:47:54.888705 8021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:47:54.899590 8021 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:47:54.900800 8021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:47:54.913260 8021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Generated new TSK 0
I20260927 22:47:54.914047 8021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 22:47:54.963078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43877
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:55.270506 8035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:55.270776 8035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:55.270880 8035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:55.278059 8035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:55.278158 8035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:55.278244 8035 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:55.278326 8035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:47:55.287655 8035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:55.288820 8035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:55.290076 8035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:55.302045 8040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:55.302071 8043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:55.302006 8041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:55.303083 8035 server_base.cc:1034] running on GCE node
I20260927 22:47:55.303687 8035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:55.304546 8035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:55.305756 8035 hybrid_clock.cc:648] HybridClock initialized: now 1790549275305694 us; error 39 us; skew 500 ppm
I20260927 22:47:55.306032 8035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:55 dist-test-slave-kr25 krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:55.308992 8035 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:47:55.310390 8035 webserver.cc:460] Webserver started at http://127.0.97.1:42005/ using document root <none> and password file <none>
I20260927 22:47:55.311074 8035 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:55.311168 8035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:55.311498 8035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:55.314433 8035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "9577120fc7174b6d8f9169bc9a85c1e1"
format_stamp: "Formatted at 2026-09-27 22:47:55 on dist-test-slave-kr25"
server_key: "644f2a5be72886e5f334c92f752983fb"
server_key_iv: "5146dd23a4ae3072f5b2254429fe2e16"
server_key_version: "encryptionkey@0"
I20260927 22:47:55.315179 8035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9577120fc7174b6d8f9169bc9a85c1e1"
format_stamp: "Formatted at 2026-09-27 22:47:55 on dist-test-slave-kr25"
server_key: "644f2a5be72886e5f334c92f752983fb"
server_key_iv: "5146dd23a4ae3072f5b2254429fe2e16"
server_key_version: "encryptionkey@0"
I20260927 22:47:55.320128 8035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:47:55.323293 8050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:55.324661 8035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:47:55.324815 8035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "9577120fc7174b6d8f9169bc9a85c1e1"
format_stamp: "Formatted at 2026-09-27 22:47:55 on dist-test-slave-kr25"
server_key: "644f2a5be72886e5f334c92f752983fb"
server_key_iv: "5146dd23a4ae3072f5b2254429fe2e16"
server_key_version: "encryptionkey@0"
I20260927 22:47:55.325001 8035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:55.334882 8035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:55.338387 8035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:55.338644 8035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:55.341048 8035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:55.341127 8035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:55.341267 8035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:55.341369 8035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 22:47:55 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:55.361756 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.341445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44779 (local address 127.0.97.62:43877)
0927 22:47:55.341624 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:55.341630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:55.342868 (+ 1238us) server_negotiation.cc:407] Connection header received
0927 22:47:55.344918 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:55.344923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:55.345002 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:55.345087 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:55.347015 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.348128 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.350359 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.350908 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.351025 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.355561 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:55.355568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:55.355573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:55.355615 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:55.358463 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.359315 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.359321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.359325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.359500 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.360045 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.360051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.360055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.360410 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:55.360533 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:55.361220 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 22:47:55.361434 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:47:55.362831 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.341937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43877 (local address 127.0.97.1:44779)
0927 22:47:55.342688 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:55.342732 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:55.344664 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:55.345345 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:55.345358 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:55.346082 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:55.346827 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.346855 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.348290 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.348295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.350233 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.350245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.351143 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.351151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.351358 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.352167 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:55.352207 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:55.355283 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:55.358611 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.358621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.358636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.359192 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.359628 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.359633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.359637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.359938 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.360533 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:55.360541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:55.360881 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 22:47:55.361126 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 22:47:55.361618 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":308,"threads_started":1}
I20260927 22:47:55.383129 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.369184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52317 (local address 127.0.97.62:43877)
0927 22:47:55.369356 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:55.369361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:55.370064 (+ 703us) server_negotiation.cc:407] Connection header received
0927 22:47:55.370242 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:55.370246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:55.370303 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:55.370369 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:55.371530 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.372248 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.374224 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.374671 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.374810 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.376893 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:55.376900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:55.376905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:55.376957 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:55.380178 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.381020 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.381026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.381030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.381197 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.381654 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.381660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.381664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.382047 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:55.382203 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:55.382710 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 22:47:55.382898 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:47:55.383535 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.369236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43877 (local address 127.0.97.1:52317)
0927 22:47:55.369910 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:55.369974 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:55.370128 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:55.370411 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:55.370415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:55.370995 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:55.371393 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.371405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.372371 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.372377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.374095 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.374111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.374884 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.374891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.375088 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.375790 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:55.375815 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:55.376729 (+ 914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:55.380338 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.380345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.380349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.380903 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.381355 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.381362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.381366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.381526 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.382171 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:55.382177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:55.382396 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 22:47:55.382581 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 22:47:55.382994 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":355,"threads_started":1}
I20260927 22:47:55.390180 8035 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40223
I20260927 22:47:55.390201 8169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40223 every 8 connection(s)
I20260927 22:47:55.391333 8035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:47:55.391850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 8035
I20260927 22:47:55.392027 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:47:55.392764 388 external_mini_cluster.cc:1428] Setting key 4e650071cd02accfd91ee3055f03a9d1
I20260927 22:47:55.393497 8170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43877
I20260927 22:47:55.393714 8170 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:55.394356 8170 heartbeater.cc:511] Master 127.0.97.62:43877 requested a full tablet report, sending...
I20260927 22:47:55.397325 7964 ts_manager.cc:194] Registered new tserver with Master: 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223)
I20260927 22:47:55.398693 7964 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44779
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 22:47:55.434388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43877
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:55.738440 8174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:55.738777 8174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:55.738914 8174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:55.745944 8174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:55.746068 8174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:55.746191 8174 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:55.746307 8174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:47:55.755877 8174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:55.757073 8174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:55.758352 8174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:55.770380 8179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:55.770355 8180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:55.770416 8182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:55.771550 8174 server_base.cc:1034] running on GCE node
I20260927 22:47:55.772050 8174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:55.772872 8174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:55.774098 8174 hybrid_clock.cc:648] HybridClock initialized: now 1790549275774041 us; error 37 us; skew 500 ppm
I20260927 22:47:55.774379 8174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:55 dist-test-slave-kr25 krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:55.777247 8174 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:47:55.778508 8174 webserver.cc:460] Webserver started at http://127.0.97.2:37841/ using document root <none> and password file <none>
I20260927 22:47:55.779135 8174 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:55.779237 8174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:55.779567 8174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:55.782289 8174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "6da16e51fc9347849f49280001f80f1b"
format_stamp: "Formatted at 2026-09-27 22:47:55 on dist-test-slave-kr25"
server_key: "81661784f2efff36708b811797ddf98b"
server_key_iv: "25be2e8658df12008f3fed5416fd5298"
server_key_version: "encryptionkey@0"
I20260927 22:47:55.783008 8174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6da16e51fc9347849f49280001f80f1b"
format_stamp: "Formatted at 2026-09-27 22:47:55 on dist-test-slave-kr25"
server_key: "81661784f2efff36708b811797ddf98b"
server_key_iv: "25be2e8658df12008f3fed5416fd5298"
server_key_version: "encryptionkey@0"
I20260927 22:47:55.787925 8174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:47:55.791196 8189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:55.792474 8174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:47:55.792621 8174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "6da16e51fc9347849f49280001f80f1b"
format_stamp: "Formatted at 2026-09-27 22:47:55 on dist-test-slave-kr25"
server_key: "81661784f2efff36708b811797ddf98b"
server_key_iv: "25be2e8658df12008f3fed5416fd5298"
server_key_version: "encryptionkey@0"
I20260927 22:47:55.792812 8174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:55.814030 8174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:55.818006 8174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:55.818230 8174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:55.820782 8174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:55.820863 8174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:55.820992 8174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:55.821060 8174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:47:55 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:55.842157 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.821079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40487 (local address 127.0.97.62:43877)
0927 22:47:55.821258 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:55.821264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:55.822576 (+ 1312us) server_negotiation.cc:407] Connection header received
0927 22:47:55.824671 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:55.824676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:55.824767 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:55.824878 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:55.827068 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.828246 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.830116 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.830742 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.830867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.835571 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:55.835578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:55.835583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:55.835628 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:55.838998 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.839947 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.839953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.839957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.840075 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.840486 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.840491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.840494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.840820 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:55.840965 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:55.841595 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 22:47:55.841819 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:47:55.843122 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.821588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43877 (local address 127.0.97.2:40487)
0927 22:47:55.822391 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:55.822454 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:55.824384 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:55.825147 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:55.825161 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:55.825930 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:55.826851 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.826882 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.828449 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.828458 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.829983 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.829995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.830980 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.830986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.831173 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.831856 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:55.831881 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:55.835250 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:55.839162 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.839173 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.839190 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.839838 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.840174 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.840179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.840183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.840368 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.840945 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:55.840952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:55.841266 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 22:47:55.841492 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 22:47:55.841979 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":330,"threads_started":1}
I20260927 22:47:55.859676 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.847993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43727 (local address 127.0.97.62:43877)
0927 22:47:55.848142 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:55.848146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:55.848767 (+ 621us) server_negotiation.cc:407] Connection header received
0927 22:47:55.848910 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:55.848915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:55.848977 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:55.849054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:55.850133 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.851083 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.852531 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.852959 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.853060 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.854630 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:55.854635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:55.854639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:55.854665 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:55.857295 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.857862 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.857868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.857872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.857980 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:55.858288 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:55.858293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:55.858297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:55.858601 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:55.858733 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:55.859124 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 22:47:55.859317 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 22:47:55.859810 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:55.847942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43877 (local address 127.0.97.2:43727)
0927 22:47:55.848678 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:55.848695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:55.848809 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:55.849085 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:55.849089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:55.849590 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:55.849965 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.849979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.851218 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.851221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.852405 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:55.852417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:55.853121 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:55.853127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:55.853293 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:55.853820 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:55.853837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:55.854466 (+ 629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:55.857408 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.857414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.857417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.857761 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.858074 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:55.858079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:55.858083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:55.858198 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:55.858708 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:55.858713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:55.858888 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 22:47:55.859032 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:47:55.859345 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":460,"threads_started":1}
I20260927 22:47:55.874212 8174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44975
I20260927 22:47:55.874233 8308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44975 every 8 connection(s)
I20260927 22:47:55.875377 8174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:47:55.877310 8309 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43877
I20260927 22:47:55.877509 8309 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:55.878126 8309 heartbeater.cc:511] Master 127.0.97.62:43877 requested a full tablet report, sending...
I20260927 22:47:55.879336 7964 ts_manager.cc:194] Registered new tserver with Master: 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975)
I20260927 22:47:55.880152 7964 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40487
I20260927 22:47:55.881528 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 8174
I20260927 22:47:55.881685 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:47:55.882210 388 external_mini_cluster.cc:1428] Setting key ab4c3daed8c5d51c5aa1ab3dbdf7d3a1
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:47:55.923382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43877
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:56.226346 8313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:56.226621 8313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:56.226764 8313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:56.234033 8313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:56.234145 8313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:56.234262 8313 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:47:56.234375 8313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:47:56.244009 8313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:56.245298 8313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:56.246573 8313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:56.258399 8318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:56.258484 8321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:56.258422 8319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:56.259127 8313 server_base.cc:1034] running on GCE node
I20260927 22:47:56.259759 8313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:56.260599 8313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:56.261798 8313 hybrid_clock.cc:648] HybridClock initialized: now 1790549276261740 us; error 30 us; skew 500 ppm
I20260927 22:47:56.262070 8313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:47:56 dist-test-slave-kr25 krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:47:56.269048 8313 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:47:56.270376 8313 webserver.cc:460] Webserver started at http://127.0.97.3:43393/ using document root <none> and password file <none>
I20260927 22:47:56.271034 8313 fs_manager.cc:359] Metadata directory not provided
I20260927 22:47:56.271116 8313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:47:56.271369 8313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:47:56.274309 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "559c805f26eb4d23a7aa119364e42de4"
format_stamp: "Formatted at 2026-09-27 22:47:56 on dist-test-slave-kr25"
server_key: "fe2826552b9de74f345c2e95b9fda305"
server_key_iv: "0ac992a4657ea5b323174a9b9ea4c582"
server_key_version: "encryptionkey@0"
I20260927 22:47:56.275025 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "559c805f26eb4d23a7aa119364e42de4"
format_stamp: "Formatted at 2026-09-27 22:47:56 on dist-test-slave-kr25"
server_key: "fe2826552b9de74f345c2e95b9fda305"
server_key_iv: "0ac992a4657ea5b323174a9b9ea4c582"
server_key_version: "encryptionkey@0"
I20260927 22:47:56.279840 8313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:47:56.282898 8328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:56.284133 8313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:47:56.284269 8313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "559c805f26eb4d23a7aa119364e42de4"
format_stamp: "Formatted at 2026-09-27 22:47:56 on dist-test-slave-kr25"
server_key: "fe2826552b9de74f345c2e95b9fda305"
server_key_iv: "0ac992a4657ea5b323174a9b9ea4c582"
server_key_version: "encryptionkey@0"
I20260927 22:47:56.284446 8313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:47:56.303738 8313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:47:56.308602 8313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:47:56.308826 8313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:47:56.311321 8313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:47:56.311408 8313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:47:56.311554 8313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:47:56.311612 8313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:47:56 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:56.333707 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.311891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51357 (local address 127.0.97.62:43877)
0927 22:47:56.312104 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.312110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.313478 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 22:47:56.316087 (+ 2609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.316093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.316180 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.316281 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:56.318398 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.319252 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.321986 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.322382 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.322509 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.326906 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:56.326911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:56.326914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:56.326945 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:56.330295 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.331200 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.331204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.331206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.331322 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.331758 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.331763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.331766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.332022 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:56.332161 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.333106 (+ 945us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.333335 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:47:56.335296 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.312470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43877 (local address 127.0.97.3:51357)
0927 22:47:56.313307 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.313352 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.315628 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.316608 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.316622 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.317371 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:56.318207 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.318232 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.319469 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.319476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.321829 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.321846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.322653 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.322661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.322960 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.323733 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:56.323766 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:56.326662 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:56.330432 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:56.330442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:56.330469 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:56.331033 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:56.331422 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:56.331427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:56.331449 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:56.331650 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:56.332237 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:56.332245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:56.332666 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.333016 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.333592 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":324,"threads_started":1}
I20260927 22:47:56.354485 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.340958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57001 (local address 127.0.97.62:43877)
0927 22:47:56.341097 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.341100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.341665 (+ 565us) server_negotiation.cc:407] Connection header received
0927 22:47:56.341842 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.341847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.341926 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.342020 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:56.343224 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.344224 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.346151 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.346559 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.346652 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.348935 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:56.348942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:56.348947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:56.348985 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:56.351628 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.352546 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.352552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.352555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.352741 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.353126 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.353131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.353135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.353399 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:56.353547 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.353911 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.354101 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:47:56.354554 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.340985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43877 (local address 127.0.97.3:57001)
0927 22:47:56.341573 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.341588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.341727 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.342077 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.342085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.342714 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:56.343087 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.343100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.344359 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.344366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.346007 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.346023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.346726 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.346733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.346905 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.347552 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:56.347575 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:56.348749 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:56.351740 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:56.351746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:56.351753 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:56.352376 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:56.352886 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:56.352891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:56.352895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:56.353019 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:56.353517 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:56.353522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:56.353688 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.353802 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.354117 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":331,"threads_started":1}
I20260927 22:47:56.360510 8313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44217
I20260927 22:47:56.360531 8447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44217 every 8 connection(s)
I20260927 22:47:56.361629 8313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:47:56.362972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 8313
I20260927 22:47:56.363133 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:47:56.363504 8448 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43877
I20260927 22:47:56.363689 388 external_mini_cluster.cc:1428] Setting key d4020c7f01b7cd651e7604bf93d7892f
I20260927 22:47:56.363847 8448 heartbeater.cc:464] Registering TS with master...
I20260927 22:47:56.364537 8448 heartbeater.cc:511] Master 127.0.97.62:43877 requested a full tablet report, sending...
I20260927 22:47:56.366178 7963 ts_manager.cc:194] Registered new tserver with Master: 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:56.367110 7963 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51357
I20260927 22:47:56.378777 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:47:56.401294 8170 heartbeater.cc:503] Master 127.0.97.62:43877 was elected leader, sending a full tablet report...
Sep 27 22:47:56 dist-test-slave-kr25 krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549276, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:47:56 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:47:56 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:47:56.423642 8012 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:56.414072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.97.62:43877)
0927 22:47:56.414228 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.414233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.414480 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:47:56.414674 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.414678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.414747 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.414835 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:56.416121 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.416875 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.418043 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.418454 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.418547 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.423407 (+ 4860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51996: BlockingRecv error: recv got EOF from 127.0.0.1:51996 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 22:47:56 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549276, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:56.438756 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.426476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.0.97.62:43877)
0927 22:47:56.426606 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.426609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.426931 (+ 322us) server_negotiation.cc:407] Connection header received
0927 22:47:56.427072 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.427076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.427133 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.427211 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:56.428513 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.429312 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.430446 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.430903 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.431011 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.434370 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:56.434375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:56.434378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:56.434410 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:56.436773 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.437303 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.437307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.437309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.437439 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.437697 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.437700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.437702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.437950 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:56.438152 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.438353 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.438522 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:47:56.442641 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:56.449397 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:56.477758 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.461422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.0.97.3:44217)
0927 22:47:56.469615 (+ 8193us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.469626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.469665 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:47:56.469774 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.469783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.470093 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.470310 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:56.471136 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.472700 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.475136 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.476604 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.476728 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.477018 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.477254 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.477427 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7965,"thread_start_us":8300,"threads_started":1}
I20260927 22:47:56.477770 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.461107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44217 (local address 127.0.0.1:39474)
0927 22:47:56.468828 (+ 7721us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.468848 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.468972 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.470354 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.470363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.470400 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:56.470900 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.470914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.472856 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.472861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.474960 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.474973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.476779 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.476785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.476971 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.477074 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.477203 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.477422 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7584,"thread_start_us":9924,"threads_started":1}
I20260927 22:47:56.478464 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.460710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.0.97.1:40223)
0927 22:47:56.470987 (+ 10277us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.470996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.471024 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:47:56.471111 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.471118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.471328 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.472074 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:56.472231 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.473703 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.475938 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.477246 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.477369 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.477547 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.478048 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.478201 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10116,"thread_start_us":9990,"threads_started":1}
I20260927 22:47:56.478528 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.460572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40223 (local address 127.0.0.1:55262)
0927 22:47:56.467361 (+ 6789us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.467406 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.467635 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.471555 (+ 3920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.471560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.471585 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:56.471915 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.471928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.473833 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.473838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.475767 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.475779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.477672 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.477678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.477875 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.477910 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.477994 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.478123 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6513,"mutex_wait_us":3,"thread_start_us":6350,"threads_started":1}
I20260927 22:47:56.478842 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.461921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44975 (local address 127.0.0.1:44022)
0927 22:47:56.471273 (+ 9352us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.471290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.471416 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.472227 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.472231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.472253 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:56.472627 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.472638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.474238 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.474243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.476215 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.476229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.478080 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.478085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.478249 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.478279 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.478354 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.478496 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9240,"mutex_wait_us":37,"thread_start_us":6642,"threads_started":1}
I20260927 22:47:56.479482 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.462448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.0.97.2:44975)
0927 22:47:56.468354 (+ 5906us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.468366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.471404 (+ 3038us) server_negotiation.cc:407] Connection header received
0927 22:47:56.471583 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.471590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.471803 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.471997 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:56.472824 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.474116 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.476389 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.477644 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.477759 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.477951 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.479035 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.479199 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5659,"thread_start_us":5512,"threads_started":1}
I20260927 22:47:56.481907 8100 tablet_service.cc:1467] Processing CreateTablet for tablet 490a813a08084e469982ff77f21a6318 (DEFAULT_TABLE table=test-table [id=4529da4f38964487b4c17781f2832f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:56.481999 8378 tablet_service.cc:1467] Processing CreateTablet for tablet 490a813a08084e469982ff77f21a6318 (DEFAULT_TABLE table=test-table [id=4529da4f38964487b4c17781f2832f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:56.482250 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 490a813a08084e469982ff77f21a6318 (DEFAULT_TABLE table=test-table [id=4529da4f38964487b4c17781f2832f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:56.483057 8100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490a813a08084e469982ff77f21a6318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.483338 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490a813a08084e469982ff77f21a6318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.483351 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490a813a08084e469982ff77f21a6318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.493868 8472 tablet_bootstrap.cc:492] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Bootstrap starting.
I20260927 22:47:56.497277 8472 tablet_bootstrap.cc:654] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.497643 8474 tablet_bootstrap.cc:492] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Bootstrap starting.
I20260927 22:47:56.499014 8472 log.cc:824] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:56.499660 8473 tablet_bootstrap.cc:492] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Bootstrap starting.
I20260927 22:47:56.501104 8474 tablet_bootstrap.cc:654] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.502043 8473 tablet_bootstrap.cc:654] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.502357 8474 log.cc:824] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:56.502988 8473 log.cc:824] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:47:56.505412 8474 tablet_bootstrap.cc:492] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: No bootstrap required, opened a new log
I20260927 22:47:56.505684 8474 ts_tablet_manager.cc:1397] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260927 22:47:56.505965 8472 tablet_bootstrap.cc:492] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: No bootstrap required, opened a new log
I20260927 22:47:56.506225 8472 ts_tablet_manager.cc:1397] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260927 22:47:56.507571 8473 tablet_bootstrap.cc:492] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: No bootstrap required, opened a new log
I20260927 22:47:56.507812 8473 ts_tablet_manager.cc:1397] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260927 22:47:56.514385 8473 raft_consensus.cc:348] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.514739 8473 raft_consensus.cc:374] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.514832 8473 raft_consensus.cc:729] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.515390 8473 consensus_queue.cc:260] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.515826 8474 raft_consensus.cc:348] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.516239 8474 raft_consensus.cc:374] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.516116 8472 raft_consensus.cc:348] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.516403 8474 raft_consensus.cc:729] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c805f26eb4d23a7aa119364e42de4, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.516506 8472 raft_consensus.cc:374] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.516580 8472 raft_consensus.cc:729] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9577120fc7174b6d8f9169bc9a85c1e1, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.516589 8309 heartbeater.cc:503] Master 127.0.97.62:43877 was elected leader, sending a full tablet report...
I20260927 22:47:56.517202 8473 ts_tablet_manager.cc:1428] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260927 22:47:56.517088 8472 consensus_queue.cc:260] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.517091 8474 consensus_queue.cc:260] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.518647 8474 ts_tablet_manager.cc:1428] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260927 22:47:56.518946 8472 ts_tablet_manager.cc:1428] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260927 22:47:56.519790 8448 heartbeater.cc:503] Master 127.0.97.62:43877 was elected leader, sending a full tablet report...
W20260927 22:47:56.613127 8449 tablet.cc:2369] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:47:56.627178 8310 tablet.cc:2369] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:56.631377 8478 raft_consensus.cc:482] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:56.631716 8478 raft_consensus.cc:504] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.633381 8478 leader_election.cc:288] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:56.640841 8479 raft_consensus.cc:482] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:56.641129 8479 raft_consensus.cc:504] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.642282 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.633729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60847 (local address 127.0.97.1:40223)
0927 22:47:56.633920 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.633937 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.634677 (+ 740us) server_negotiation.cc:407] Connection header received
0927 22:47:56.634845 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.634849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.634951 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.635030 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:56.635770 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.636880 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.639662 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.640934 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.641087 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.641219 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.641931 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.642059 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:47:56.642585 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.633800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44217 (local address 127.0.97.2:50403)
0927 22:47:56.634583 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.634598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.634769 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.635224 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.635229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.635264 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:56.635780 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.635791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.637075 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.637080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.639481 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.639495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.641420 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.641427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.641691 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.641774 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.641873 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.642050 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":136,"spinlock_wait_cycles":24320,"thread_start_us":538,"threads_started":1}
I20260927 22:47:56.642609 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.633596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40223 (local address 127.0.97.2:60847)
0927 22:47:56.634562 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.634583 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.634718 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.635114 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.635119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.635142 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:56.635611 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.635623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.636993 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.636997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.639483 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.639495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.641410 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.641417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.641717 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.641779 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.641870 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.642050 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":177,"thread_start_us":508,"threads_started":1}
I20260927 22:47:56.642885 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.633877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50403 (local address 127.0.97.3:44217)
0927 22:47:56.634069 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.634075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.634685 (+ 610us) server_negotiation.cc:407] Connection header received
0927 22:47:56.634915 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.634919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.634986 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.635061 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:56.635958 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.636964 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.639660 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.641007 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.641133 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.641242 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.642531 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.642655 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:47:56.642930 8479 leader_election.cc:288] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975)
I20260927 22:47:56.643918 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490a813a08084e469982ff77f21a6318" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4" is_pre_election: true
I20260927 22:47:56.644320 8398 raft_consensus.cc:2454] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 0.
W20260927 22:47:56.643882 8171 tablet.cc:2369] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:47:56.645326 8191 leader_election.cc:302] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4, 6da16e51fc9347849f49280001f80f1b; no voters:
I20260927 22:47:56.645781 8478 raft_consensus.cc:2781] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:56.645618 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490a813a08084e469982ff77f21a6318" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" is_pre_election: true
I20260927 22:47:56.645943 8478 raft_consensus.cc:482] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:56.646056 8478 raft_consensus.cc:3037] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:56.646171 8120 raft_consensus.cc:2454] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 0.
I20260927 22:47:56.649400 8478 raft_consensus.cc:504] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.650101 8478 leader_election.cc:288] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 election: Requested vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:56.650527 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490a813a08084e469982ff77f21a6318" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4"
I20260927 22:47:56.650695 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490a813a08084e469982ff77f21a6318" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1"
I20260927 22:47:56.650775 8398 raft_consensus.cc:3037] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:56.650902 8120 raft_consensus.cc:3037] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:56.651850 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.643391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43867 (local address 127.0.97.2:44975)
0927 22:47:56.644783 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.644789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.644805 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:47:56.644855 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.644860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.644918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.644997 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:56.645678 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.646615 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.649086 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.650368 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.650509 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.650606 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.651510 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.651632 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1279}
I20260927 22:47:56.652050 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.643337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44975 (local address 127.0.97.3:43867)
0927 22:47:56.644168 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.644187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.644282 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.645299 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.645303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.645329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:56.645504 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.645515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.646759 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.646765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.648856 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.648872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.650690 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.650695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.650882 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.650916 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.651003 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.651141 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713}
I20260927 22:47:56.653048 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.644157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50865 (local address 127.0.97.1:40223)
0927 22:47:56.646783 (+ 2626us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.646789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.646803 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:47:56.646861 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.646865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.646937 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.647011 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:56.647753 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.648415 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.650147 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.651633 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.651746 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.651842 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.652635 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.652766 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":623,"threads_started":1}
I20260927 22:47:56.653048 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.643428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40223 (local address 127.0.97.3:50865)
0927 22:47:56.646034 (+ 2606us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.646052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.646149 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.647234 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.647237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.647255 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:56.647612 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.647622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.648538 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.648542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.649853 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.649864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.652260 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.652267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.652462 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.652494 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.652593 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.652711 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2472,"thread_start_us":3614,"threads_started":1}
I20260927 22:47:56.653745 8398 raft_consensus.cc:2454] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 1.
I20260927 22:47:56.653882 8120 raft_consensus.cc:2454] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 1.
I20260927 22:47:56.654181 8191 leader_election.cc:302] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4, 6da16e51fc9347849f49280001f80f1b; no voters:
I20260927 22:47:56.654440 8478 raft_consensus.cc:2781] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:56.654924 8478 raft_consensus.cc:686] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 1 LEADER]: Becoming Leader. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Running, Role: LEADER
I20260927 22:47:56.655035 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490a813a08084e469982ff77f21a6318" candidate_uuid: "559c805f26eb4d23a7aa119364e42de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da16e51fc9347849f49280001f80f1b" is_pre_election: true
I20260927 22:47:56.655236 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490a813a08084e469982ff77f21a6318" candidate_uuid: "559c805f26eb4d23a7aa119364e42de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" is_pre_election: true
I20260927 22:47:56.655491 8120 raft_consensus.cc:2379] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 559c805f26eb4d23a7aa119364e42de4 in current term 1: Already voted for candidate 6da16e51fc9347849f49280001f80f1b in this term.
I20260927 22:47:56.655483 8478 consensus_queue.cc:237] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } }
I20260927 22:47:56.656472 8329 leader_election.cc:302] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4; no voters: 6da16e51fc9347849f49280001f80f1b, 9577120fc7174b6d8f9169bc9a85c1e1
I20260927 22:47:56.656806 8479 raft_consensus.cc:2726] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:47:56.660025 7963 catalog_manager.cc:5495] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6da16e51fc9347849f49280001f80f1b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6da16e51fc9347849f49280001f80f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } health_report { overall_health: HEALTHY } } }
I20260927 22:47:56.716250 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.711497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.0.97.2:44975)
0927 22:47:56.711691 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.711697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.711777 (+ 80us) server_negotiation.cc:407] Connection header received
0927 22:47:56.712006 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.712010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.712101 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.712229 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:56.712499 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.713205 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.714385 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.714767 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.714853 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.715174 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:56.715726 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:56.715835 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.715899 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.715968 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:47:56.723762 8120 raft_consensus.cc:1261] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Refusing update from remote peer 6da16e51fc9347849f49280001f80f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:56.723820 8398 raft_consensus.cc:1261] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Refusing update from remote peer 6da16e51fc9347849f49280001f80f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:56.724627 8485 consensus_queue.cc:1035] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:56.725013 8478 consensus_queue.cc:1035] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:56.752190 7964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:56.752414 7964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:56.755645 7964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Sending DeleteTablet for 3 replicas of tablet 490a813a08084e469982ff77f21a6318
I20260927 22:47:56.756630 8100 tablet_service.cc:1514] Processing DeleteTablet for tablet 490a813a08084e469982ff77f21a6318 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:56 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55262
I20260927 22:47:56.756673 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet 490a813a08084e469982ff77f21a6318 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:56 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39474
I20260927 22:47:56.756680 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet 490a813a08084e469982ff77f21a6318 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:56 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44022
I20260927 22:47:56.758716 7964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:56.759827 7964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:56.768637 8377 tablet_service.cc:1467] Processing CreateTablet for tablet 553d5ea6c4db4245b8d7e84619ba3c7b (DEFAULT_TABLE table=test-table [id=b71205608cfa420ab9658e38eab01d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:56.769196 8377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d5ea6c4db4245b8d7e84619ba3c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.769812 8498 tablet_replica.cc:330] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: stopping tablet replica
I20260927 22:47:56.770274 8498 raft_consensus.cc:2229] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:56.770944 8498 raft_consensus.cc:2258] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:56.771782 8499 tablet_replica.cc:330] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: stopping tablet replica
I20260927 22:47:56.772219 8499 raft_consensus.cc:2229] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:56.772795 8499 raft_consensus.cc:2258] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:56.772962 8099 tablet_service.cc:1467] Processing CreateTablet for tablet 553d5ea6c4db4245b8d7e84619ba3c7b (DEFAULT_TABLE table=test-table [id=b71205608cfa420ab9658e38eab01d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:56.773401 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d5ea6c4db4245b8d7e84619ba3c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.775759 8498 ts_tablet_manager.cc:1905] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:56.780644 8499 ts_tablet_manager.cc:1905] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:56.782622 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 553d5ea6c4db4245b8d7e84619ba3c7b (DEFAULT_TABLE table=test-table [id=b71205608cfa420ab9658e38eab01d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:56.787938 8500 tablet_replica.cc:330] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: stopping tablet replica
I20260927 22:47:56.788203 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d5ea6c4db4245b8d7e84619ba3c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.788384 8500 raft_consensus.cc:2229] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:56.788888 8500 raft_consensus.cc:2258] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:56.790012 8499 ts_tablet_manager.cc:1918] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:56.790182 8499 log.cc:1196] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/490a813a08084e469982ff77f21a6318
I20260927 22:47:56.790688 8499 ts_tablet_manager.cc:1939] T 490a813a08084e469982ff77f21a6318 P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting consensus metadata
I20260927 22:47:56.790990 8500 ts_tablet_manager.cc:1905] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:56.791507 8472 tablet_bootstrap.cc:492] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: Bootstrap starting.
I20260927 22:47:56.792462 7951 catalog_manager.cc:4841] TS 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223): tablet 490a813a08084e469982ff77f21a6318 (table test-table [id=4529da4f38964487b4c17781f2832f79]) successfully deleted
I20260927 22:47:56.792541 8498 ts_tablet_manager.cc:1918] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:56.792814 8498 log.cc:1196] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/490a813a08084e469982ff77f21a6318
I20260927 22:47:56.793049 8472 tablet_bootstrap.cc:654] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.793717 8498 ts_tablet_manager.cc:1939] T 490a813a08084e469982ff77f21a6318 P 6da16e51fc9347849f49280001f80f1b: Deleting consensus metadata
I20260927 22:47:56.795522 7948 catalog_manager.cc:4841] TS 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975): tablet 490a813a08084e469982ff77f21a6318 (table test-table [id=4529da4f38964487b4c17781f2832f79]) successfully deleted
I20260927 22:47:56.797533 8474 tablet_bootstrap.cc:492] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: Bootstrap starting.
I20260927 22:47:56.799052 8474 tablet_bootstrap.cc:654] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.800433 8500 ts_tablet_manager.cc:1918] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:56.800573 8500 log.cc:1196] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/490a813a08084e469982ff77f21a6318
I20260927 22:47:56.801020 8500 ts_tablet_manager.cc:1939] T 490a813a08084e469982ff77f21a6318 P 559c805f26eb4d23a7aa119364e42de4: Deleting consensus metadata
I20260927 22:47:56.801988 8473 tablet_bootstrap.cc:492] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: Bootstrap starting.
I20260927 22:47:56.802000 8474 tablet_bootstrap.cc:492] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: No bootstrap required, opened a new log
I20260927 22:47:56.802124 8474 ts_tablet_manager.cc:1397] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 22:47:56.802858 8474 raft_consensus.cc:348] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.803078 8474 raft_consensus.cc:374] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.803160 8474 raft_consensus.cc:729] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c805f26eb4d23a7aa119364e42de4, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.803424 8474 consensus_queue.cc:260] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.803697 8473 tablet_bootstrap.cc:654] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.806125 8474 ts_tablet_manager.cc:1428] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 22:47:56.807879 8472 tablet_bootstrap.cc:492] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: No bootstrap required, opened a new log
I20260927 22:47:56.808020 8472 ts_tablet_manager.cc:1397] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260927 22:47:56.808281 7949 catalog_manager.cc:4841] TS 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217): tablet 490a813a08084e469982ff77f21a6318 (table test-table [id=4529da4f38964487b4c17781f2832f79]) successfully deleted
I20260927 22:47:56.808816 8472 raft_consensus.cc:348] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.809023 8472 raft_consensus.cc:374] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.809111 8472 raft_consensus.cc:729] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9577120fc7174b6d8f9169bc9a85c1e1, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.809369 8472 consensus_queue.cc:260] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.809684 8473 tablet_bootstrap.cc:492] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: No bootstrap required, opened a new log
I20260927 22:47:56.809831 8473 ts_tablet_manager.cc:1397] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 22:47:56.809849 8472 ts_tablet_manager.cc:1428] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:47:56.810585 8473 raft_consensus.cc:348] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.810802 8473 raft_consensus.cc:374] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.810900 8473 raft_consensus.cc:729] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.811184 8473 consensus_queue.cc:260] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.811785 8473 ts_tablet_manager.cc:1428] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:47:56.865217 8493 raft_consensus.cc:482] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:56.865445 8493 raft_consensus.cc:504] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.866163 8493 leader_election.cc:288] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:56.866720 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d5ea6c4db4245b8d7e84619ba3c7b" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" is_pre_election: true
I20260927 22:47:56.866705 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d5ea6c4db4245b8d7e84619ba3c7b" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4" is_pre_election: true
I20260927 22:47:56.866953 8120 raft_consensus.cc:2454] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 0.
I20260927 22:47:56.866954 8398 raft_consensus.cc:2454] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 0.
I20260927 22:47:56.867400 8191 leader_election.cc:302] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4, 6da16e51fc9347849f49280001f80f1b; no voters:
I20260927 22:47:56.867723 8493 raft_consensus.cc:2781] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:56.867832 8493 raft_consensus.cc:482] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:56.867923 8493 raft_consensus.cc:3037] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:56.870090 8493 raft_consensus.cc:504] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.870599 8493 leader_election.cc:288] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 election: Requested vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:56.871068 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d5ea6c4db4245b8d7e84619ba3c7b" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1"
I20260927 22:47:56.871101 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d5ea6c4db4245b8d7e84619ba3c7b" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4"
I20260927 22:47:56.871240 8120 raft_consensus.cc:3037] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:56.871335 8398 raft_consensus.cc:3037] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:56.873289 8120 raft_consensus.cc:2454] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 1.
I20260927 22:47:56.873662 8193 leader_election.cc:302] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6da16e51fc9347849f49280001f80f1b, 9577120fc7174b6d8f9169bc9a85c1e1; no voters:
I20260927 22:47:56.874017 8493 raft_consensus.cc:2781] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:56.874155 8493 raft_consensus.cc:686] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 1 LEADER]: Becoming Leader. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Running, Role: LEADER
I20260927 22:47:56.874159 8398 raft_consensus.cc:2454] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 1.
I20260927 22:47:56.874432 8493 consensus_queue.cc:237] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.877061 7962 catalog_manager.cc:5495] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6da16e51fc9347849f49280001f80f1b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6da16e51fc9347849f49280001f80f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:47:56 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:56.934794 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.921751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43877 (local address 127.0.0.1:52022)
0927 22:47:56.922412 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:56.922431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:56.922563 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:56.922851 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:56.922856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:56.923580 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:47:56.923856 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.923866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.924860 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.924865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.925803 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:56.925814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.926576 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.926583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:56.926747 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.927375 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:47:56.927396 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:47:56.929545 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:47:56.932431 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:56.932442 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:56.932450 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:56.932859 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:56.933175 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:47:56.933180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:47:56.933183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:47:56.933309 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:47:56.933794 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:47:56.933802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:47:56.933955 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 22:47:56.934089 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 22:47:56.934347 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":410,"threads_started":1}
I20260927 22:47:56.934816 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:56.921769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.0.97.62:43877)
0927 22:47:56.922016 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:56.922022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:56.922510 (+ 488us) server_negotiation.cc:407] Connection header received
0927 22:47:56.922664 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:56.922669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:56.922724 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:56.922800 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:56.923982 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.924737 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.926006 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:56.926415 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:56.926523 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:56.929710 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:56.929716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:56.929720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:56.929748 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:56.932305 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.932961 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.932966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.932970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.933081 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:56.933400 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:56.933405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:56.933408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:56.933657 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:56.933794 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:56.934186 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 22:47:56.934484 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":107}
I20260927 22:47:56.939981 7962 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52022:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:47:56.941033 7962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:56.947748 8099 tablet_service.cc:1467] Processing CreateTablet for tablet 4d206d41a2d445bf96ed64ae2ca614dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edd48954dda746c8a75cfde6000eb6fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:56.948030 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 4d206d41a2d445bf96ed64ae2ca614dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edd48954dda746c8a75cfde6000eb6fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:56.948300 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d206d41a2d445bf96ed64ae2ca614dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.948568 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d206d41a2d445bf96ed64ae2ca614dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.952742 8377 tablet_service.cc:1467] Processing CreateTablet for tablet 4d206d41a2d445bf96ed64ae2ca614dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edd48954dda746c8a75cfde6000eb6fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:47:56.953230 8377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d206d41a2d445bf96ed64ae2ca614dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:56.955369 8473 tablet_bootstrap.cc:492] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: Bootstrap starting.
I20260927 22:47:56.956315 8472 tablet_bootstrap.cc:492] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: Bootstrap starting.
I20260927 22:47:56.957089 8473 tablet_bootstrap.cc:654] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.957855 8472 tablet_bootstrap.cc:654] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.961347 8474 tablet_bootstrap.cc:492] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: Bootstrap starting.
I20260927 22:47:56.962831 8474 tablet_bootstrap.cc:654] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:56.968641 8473 tablet_bootstrap.cc:492] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: No bootstrap required, opened a new log
I20260927 22:47:56.968808 8473 ts_tablet_manager.cc:1397] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260927 22:47:56.969429 8473 raft_consensus.cc:348] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.969612 8473 raft_consensus.cc:374] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.969687 8473 raft_consensus.cc:729] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.969939 8473 consensus_queue.cc:260] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.970316 8493 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:56.970597 8493 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:56.971266 8473 ts_tablet_manager.cc:1428] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:47:56.972615 8472 tablet_bootstrap.cc:492] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: No bootstrap required, opened a new log
I20260927 22:47:56.972769 8472 ts_tablet_manager.cc:1397] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260927 22:47:56.973603 8472 raft_consensus.cc:348] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.973799 8474 tablet_bootstrap.cc:492] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: No bootstrap required, opened a new log
I20260927 22:47:56.973822 8472 raft_consensus.cc:374] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.973892 8472 raft_consensus.cc:729] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9577120fc7174b6d8f9169bc9a85c1e1, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.973942 8474 ts_tablet_manager.cc:1397] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20260927 22:47:56.974152 8472 consensus_queue.cc:260] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.974593 8474 raft_consensus.cc:348] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.974578 8480 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:56.974829 8474 raft_consensus.cc:374] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:56.974903 8480 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:56.974905 8474 raft_consensus.cc:729] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c805f26eb4d23a7aa119364e42de4, State: Initialized, Role: FOLLOWER
I20260927 22:47:56.975101 8474 consensus_queue.cc:260] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:56.975520 8479 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:56.975741 8479 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:56.975791 8472 ts_tablet_manager.cc:1428] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 22:47:56.976701 8474 ts_tablet_manager.cc:1428] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 22:47:57.238469 8479 raft_consensus.cc:482] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:57.238651 8479 raft_consensus.cc:504] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.239266 8479 leader_election.cc:288] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975)
I20260927 22:47:57.239821 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d206d41a2d445bf96ed64ae2ca614dc" candidate_uuid: "559c805f26eb4d23a7aa119364e42de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da16e51fc9347849f49280001f80f1b" is_pre_election: true
I20260927 22:47:57.239832 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d206d41a2d445bf96ed64ae2ca614dc" candidate_uuid: "559c805f26eb4d23a7aa119364e42de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" is_pre_election: true
I20260927 22:47:57.240095 8263 raft_consensus.cc:2454] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 559c805f26eb4d23a7aa119364e42de4 in term 0.
I20260927 22:47:57.240130 8120 raft_consensus.cc:2454] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 559c805f26eb4d23a7aa119364e42de4 in term 0.
I20260927 22:47:57.240561 8329 leader_election.cc:302] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4, 6da16e51fc9347849f49280001f80f1b, 9577120fc7174b6d8f9169bc9a85c1e1; no voters:
I20260927 22:47:57.240816 8479 raft_consensus.cc:2781] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:57.240923 8479 raft_consensus.cc:482] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:57.241016 8479 raft_consensus.cc:3037] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.243170 8479 raft_consensus.cc:504] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.243764 8479 leader_election.cc:288] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [CANDIDATE]: Term 1 election: Requested vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975)
I20260927 22:47:57.244123 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d206d41a2d445bf96ed64ae2ca614dc" candidate_uuid: "559c805f26eb4d23a7aa119364e42de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1"
I20260927 22:47:57.244144 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d206d41a2d445bf96ed64ae2ca614dc" candidate_uuid: "559c805f26eb4d23a7aa119364e42de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da16e51fc9347849f49280001f80f1b"
I20260927 22:47:57.244292 8263 raft_consensus.cc:3037] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.244311 8120 raft_consensus.cc:3037] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.246215 8120 raft_consensus.cc:2454] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 559c805f26eb4d23a7aa119364e42de4 in term 1.
I20260927 22:47:57.246359 8263 raft_consensus.cc:2454] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 559c805f26eb4d23a7aa119364e42de4 in term 1.
I20260927 22:47:57.246623 8332 leader_election.cc:302] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4, 9577120fc7174b6d8f9169bc9a85c1e1; no voters:
I20260927 22:47:57.246887 8479 raft_consensus.cc:2781] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:57.247231 8479 raft_consensus.cc:686] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [term 1 LEADER]: Becoming Leader. State: Replica: 559c805f26eb4d23a7aa119364e42de4, State: Running, Role: LEADER
I20260927 22:47:57.247587 8479 consensus_queue.cc:237] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.249205 8514 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: TxnStatusTablet state changed. Reason: New leader 559c805f26eb4d23a7aa119364e42de4. Latest consensus state: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:57.249476 8514 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:57.250011 8516 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:47:57.250178 8516 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:47:57.251919 7962 catalog_manager.cc:5495] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } health_report { overall_health: HEALTHY } } }
I20260927 22:47:57.271010 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.264849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44217 (local address 127.0.0.1:39490)
0927 22:47:57.265069 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:57.265113 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:57.265251 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:57.265916 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:57.265919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:57.265929 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:57.266084 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.266091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.267526 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.267530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.268537 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.268548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.269488 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.269492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.269619 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.269667 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.270623 (+ 956us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.270634 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:47:57.270679 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 22:47:57.270809 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:47:57.271130 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.264889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.0.97.3:44217)
0927 22:47:57.265519 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.265525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.265549 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:47:57.265635 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.265641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.265734 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.265904 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:57.266236 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.267393 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.268667 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.269352 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.269477 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.269846 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.270529 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.270669 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.270731 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.270808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":388,"threads_started":1}
I20260927 22:47:57.275300 8120 raft_consensus.cc:1261] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Refusing update from remote peer 559c805f26eb4d23a7aa119364e42de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.275396 8263 raft_consensus.cc:1261] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Refusing update from remote peer 559c805f26eb4d23a7aa119364e42de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.276046 8514 consensus_queue.cc:1035] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.276371 8479 consensus_queue.cc:1035] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.280020 8480 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: TxnStatusTablet state changed. Reason: New leader 559c805f26eb4d23a7aa119364e42de4. Latest consensus state: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:57.280267 8480 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:57.283819 8514 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:57.284122 8514 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:57.287683 8480 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:57.287926 8480 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 9577120fc7174b6d8f9169bc9a85c1e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:57.290715 8493 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: TxnStatusTablet state changed. Reason: New leader 559c805f26eb4d23a7aa119364e42de4. Latest consensus state: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:57.290968 8493 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:57.292654 8479 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:57.292922 8479 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 559c805f26eb4d23a7aa119364e42de4: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:47:57.307161 8493 tablet_replica.cc:439] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "559c805f26eb4d23a7aa119364e42de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } } }
I20260927 22:47:57.307413 8493 tablet_replica.cc:442] T 4d206d41a2d445bf96ed64ae2ca614dc P 6da16e51fc9347849f49280001f80f1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:47:57.318930 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.313998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56977 (local address 127.0.97.3:44217)
0927 22:47:57.314149 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.314155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.314521 (+ 366us) server_negotiation.cc:407] Connection header received
0927 22:47:57.314695 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.314699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.314759 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.314868 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:57.315253 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.316049 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.317284 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.317772 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.317884 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.318258 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.318354 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.318425 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.318553 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.318626 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:47:57.319024 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.313954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44217 (local address 127.0.97.2:56977)
0927 22:47:57.314430 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:57.314445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:57.314571 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:57.314893 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:57.314897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:57.314909 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:57.315120 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.315128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.316167 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.316173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.317156 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.317166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.317926 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.317933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.318056 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.318106 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.318442 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.318454 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:47:57.318521 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 22:47:57.318649 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":248,"threads_started":1}
I20260927 22:47:57.328244 8398 raft_consensus.cc:1261] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Refusing update from remote peer 6da16e51fc9347849f49280001f80f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.328308 8120 raft_consensus.cc:1261] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Refusing update from remote peer 6da16e51fc9347849f49280001f80f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.328807 8478 consensus_queue.cc:1035] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.329095 8493 consensus_queue.cc:1035] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.333534 8522 mvcc.cc:205] Tried to move back new op lower bound from 7334089839931940864 to 7334089838080102400. Current Snapshot: MvccSnapshot[applied={T|T < 7334089839931940864}]
I20260927 22:47:57.352314 8521 mvcc.cc:205] Tried to move back new op lower bound from 7334089839931940864 to 7334089838080102400. Current Snapshot: MvccSnapshot[applied={T|T < 7334089839931940864}]
I20260927 22:47:57.374919 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.361414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34241 (local address 127.0.97.2:44975)
0927 22:47:57.361869 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.361875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.365599 (+ 3724us) server_negotiation.cc:407] Connection header received
0927 22:47:57.365791 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.365797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.365875 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.365969 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:57.366762 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.367863 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.371846 (+ 3983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.372479 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.372572 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.374068 (+ 1496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.374204 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.374264 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.374470 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.374550 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260927 22:47:57.375702 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.361021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44975 (local address 127.0.97.3:34241)
0927 22:47:57.365483 (+ 4462us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:57.365512 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:57.365662 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:57.366167 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:57.366173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:57.366186 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:57.366585 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.366601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.368015 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.368021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.371694 (+ 3673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.371712 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.372745 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.372751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.373777 (+ 1026us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.373854 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.374339 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.374349 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:47:57.375149 (+ 800us) client_negotiation.cc:240] Negotiation successful
0927 22:47:57.375290 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4232,"thread_start_us":4094,"threads_started":1}
I20260927 22:47:57.422367 7962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:57.422607 7962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:57.425565 7962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Sending DeleteTablet for 3 replicas of tablet 553d5ea6c4db4245b8d7e84619ba3c7b
I20260927 22:47:57.426276 8099 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d5ea6c4db4245b8d7e84619ba3c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55262
I20260927 22:47:57.426314 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d5ea6c4db4245b8d7e84619ba3c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44022
I20260927 22:47:57.426676 8377 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d5ea6c4db4245b8d7e84619ba3c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39474
I20260927 22:47:57.438105 8540 tablet_replica.cc:330] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: stopping tablet replica
I20260927 22:47:57.438436 8540 raft_consensus.cc:2229] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:57.438910 8540 raft_consensus.cc:2258] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.440065 8540 ts_tablet_manager.cc:1905] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.443915 8541 tablet_replica.cc:330] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: stopping tablet replica
I20260927 22:47:57.444108 8541 raft_consensus.cc:2229] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:57.444895 8539 tablet_replica.cc:330] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: stopping tablet replica
I20260927 22:47:57.445075 8539 raft_consensus.cc:2229] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:57.445298 8539 raft_consensus.cc:2258] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.446424 8539 ts_tablet_manager.cc:1905] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.446810 8540 ts_tablet_manager.cc:1918] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:57.446919 8540 log.cc:1196] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/553d5ea6c4db4245b8d7e84619ba3c7b
I20260927 22:47:57.447304 8540 ts_tablet_manager.cc:1939] T 553d5ea6c4db4245b8d7e84619ba3c7b P 6da16e51fc9347849f49280001f80f1b: Deleting consensus metadata
I20260927 22:47:57.448298 7948 catalog_manager.cc:4841] TS 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975): tablet 553d5ea6c4db4245b8d7e84619ba3c7b (table test-table [id=b71205608cfa420ab9658e38eab01d1b]) successfully deleted
I20260927 22:47:57.450598 8541 raft_consensus.cc:2258] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.451766 8541 ts_tablet_manager.cc:1905] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.453580 8539 ts_tablet_manager.cc:1918] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:57.453799 8539 log.cc:1196] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/553d5ea6c4db4245b8d7e84619ba3c7b
I20260927 22:47:57.454279 8539 ts_tablet_manager.cc:1939] T 553d5ea6c4db4245b8d7e84619ba3c7b P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting consensus metadata
I20260927 22:47:57.455499 7951 catalog_manager.cc:4841] TS 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223): tablet 553d5ea6c4db4245b8d7e84619ba3c7b (table test-table [id=b71205608cfa420ab9658e38eab01d1b]) successfully deleted
I20260927 22:47:57.458810 8541 ts_tablet_manager.cc:1918] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:57.458928 8541 log.cc:1196] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/553d5ea6c4db4245b8d7e84619ba3c7b
I20260927 22:47:57.459270 8541 ts_tablet_manager.cc:1939] T 553d5ea6c4db4245b8d7e84619ba3c7b P 559c805f26eb4d23a7aa119364e42de4: Deleting consensus metadata
I20260927 22:47:57.460281 7949 catalog_manager.cc:4841] TS 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217): tablet 553d5ea6c4db4245b8d7e84619ba3c7b (table test-table [id=b71205608cfa420ab9658e38eab01d1b]) successfully deleted
Sep 27 22:47:57 dist-test-slave-kr25 krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:47:57 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:47:57 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:47:57.487876 8548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:47:57.476180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.0.97.62:43877)
0927 22:47:57.476676 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.476682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.477006 (+ 324us) server_negotiation.cc:407] Connection header received
0927 22:47:57.477309 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.477313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.477375 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.477492 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:57.479227 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.479965 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.481931 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.482317 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.482402 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.487570 (+ 5168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52038: BlockingRecv error: recv got EOF from 127.0.0.1:52038 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":210,"threads_started":1}
Sep 27 22:47:57 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:47:57.506114 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.490566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.0.97.62:43877)
0927 22:47:57.490712 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.490716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.491050 (+ 334us) server_negotiation.cc:407] Connection header received
0927 22:47:57.491213 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.491216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.491281 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.491355 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:57.492966 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.493636 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.495389 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.495949 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.496042 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.500345 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:57.500352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:57.500357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:57.500402 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:57.503664 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:57.504255 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:57.504261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:57.504266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:57.504421 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:57.504729 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:57.504734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:57.504737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:57.505092 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:57.505359 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.505516 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.505741 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 22:47:57.509459 7962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:57.510026 7962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:57.516009 8099 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f (DEFAULT_TABLE table=test-table [id=6d23a3b0830f44a48d29ff03d53b5db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:57.516202 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f (DEFAULT_TABLE table=test-table [id=6d23a3b0830f44a48d29ff03d53b5db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:57.516455 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:57.516605 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:57.516706 8377 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f (DEFAULT_TABLE table=test-table [id=6d23a3b0830f44a48d29ff03d53b5db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:57.517102 8377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:57.526823 8557 tablet_bootstrap.cc:492] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Bootstrap starting.
I20260927 22:47:57.528750 8557 tablet_bootstrap.cc:654] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:57.529464 8556 tablet_bootstrap.cc:492] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: Bootstrap starting.
I20260927 22:47:57.531136 8556 tablet_bootstrap.cc:654] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:57.532510 8555 tablet_bootstrap.cc:492] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: Bootstrap starting.
I20260927 22:47:57.534014 8555 tablet_bootstrap.cc:654] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:57.539372 8556 tablet_bootstrap.cc:492] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: No bootstrap required, opened a new log
I20260927 22:47:57.539532 8556 ts_tablet_manager.cc:1397] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 22:47:57.540268 8556 raft_consensus.cc:348] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.540519 8556 raft_consensus.cc:374] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:57.540607 8556 raft_consensus.cc:729] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9577120fc7174b6d8f9169bc9a85c1e1, State: Initialized, Role: FOLLOWER
I20260927 22:47:57.540841 8556 consensus_queue.cc:260] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.546350 8556 ts_tablet_manager.cc:1428] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 22:47:57.547338 8557 tablet_bootstrap.cc:492] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: No bootstrap required, opened a new log
I20260927 22:47:57.548189 8557 ts_tablet_manager.cc:1397] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.010s
I20260927 22:47:57.548640 8555 tablet_bootstrap.cc:492] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: No bootstrap required, opened a new log
I20260927 22:47:57.548784 8555 ts_tablet_manager.cc:1397] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20260927 22:47:57.549156 8557 raft_consensus.cc:348] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.549448 8557 raft_consensus.cc:374] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:57.549549 8557 raft_consensus.cc:729] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c805f26eb4d23a7aa119364e42de4, State: Initialized, Role: FOLLOWER
I20260927 22:47:57.549471 8555 raft_consensus.cc:348] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.549680 8555 raft_consensus.cc:374] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:57.549769 8555 raft_consensus.cc:729] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Initialized, Role: FOLLOWER
I20260927 22:47:57.549849 8557 consensus_queue.cc:260] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.549978 8555 consensus_queue.cc:260] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.550455 8555 ts_tablet_manager.cc:1428] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:47:57.551007 8557 ts_tablet_manager.cc:1428] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 22:47:57.621570 8478 raft_consensus.cc:482] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:57.621803 8478 raft_consensus.cc:504] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.622505 8478 leader_election.cc:288] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:57.622968 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2a979bdf2c4d9a8f668cc2ccbfa04f" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" is_pre_election: true
I20260927 22:47:57.623219 8119 raft_consensus.cc:2454] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 0.
I20260927 22:47:57.623180 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2a979bdf2c4d9a8f668cc2ccbfa04f" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4" is_pre_election: true
I20260927 22:47:57.623415 8398 raft_consensus.cc:2454] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 0.
I20260927 22:47:57.623747 8193 leader_election.cc:302] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6da16e51fc9347849f49280001f80f1b, 9577120fc7174b6d8f9169bc9a85c1e1; no voters:
I20260927 22:47:57.624061 8478 raft_consensus.cc:2781] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:57.624143 8478 raft_consensus.cc:482] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:57.624235 8478 raft_consensus.cc:3037] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.626116 8478 raft_consensus.cc:504] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.626571 8478 leader_election.cc:288] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 election: Requested vote from peers 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:57.626992 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2a979bdf2c4d9a8f668cc2ccbfa04f" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4"
I20260927 22:47:57.627024 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2a979bdf2c4d9a8f668cc2ccbfa04f" candidate_uuid: "6da16e51fc9347849f49280001f80f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9577120fc7174b6d8f9169bc9a85c1e1"
I20260927 22:47:57.627174 8398 raft_consensus.cc:3037] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.627219 8119 raft_consensus.cc:3037] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.629593 8398 raft_consensus.cc:2454] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 1.
I20260927 22:47:57.629940 8191 leader_election.cc:302] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4, 6da16e51fc9347849f49280001f80f1b; no voters:
I20260927 22:47:57.630190 8478 raft_consensus.cc:2781] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:57.630170 8119 raft_consensus.cc:2454] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da16e51fc9347849f49280001f80f1b in term 1.
I20260927 22:47:57.630344 8478 raft_consensus.cc:686] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 1 LEADER]: Becoming Leader. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Running, Role: LEADER
I20260927 22:47:57.630581 8478 consensus_queue.cc:237] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.632969 7964 catalog_manager.cc:5495] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6da16e51fc9347849f49280001f80f1b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6da16e51fc9347849f49280001f80f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:57.657912 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.652844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.0.97.2:44975)
0927 22:47:57.653014 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.653018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.653089 (+ 71us) server_negotiation.cc:407] Connection header received
0927 22:47:57.653252 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.653256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.653322 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.653393 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:57.653784 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.654511 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.656415 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.656798 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.656894 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.657347 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.657441 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.657499 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.657630 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.657688 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:47:57.660419 8119 raft_consensus.cc:1261] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Refusing update from remote peer 6da16e51fc9347849f49280001f80f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.660418 8398 raft_consensus.cc:1261] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Refusing update from remote peer 6da16e51fc9347849f49280001f80f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.660981 8493 consensus_queue.cc:1035] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.661309 8478 consensus_queue.cc:1035] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.672497 7964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:57.672719 7964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:57.676260 7964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Sending DeleteTablet for 3 replicas of tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f
I20260927 22:47:57.677060 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44022
I20260927 22:47:57.677366 8540 tablet_replica.cc:330] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: stopping tablet replica
I20260927 22:47:57.677557 8540 raft_consensus.cc:2229] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:57.677871 8540 raft_consensus.cc:2258] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.678068 8377 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39474
I20260927 22:47:57.678318 8541 tablet_replica.cc:330] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: stopping tablet replica
I20260927 22:47:57.678516 8541 raft_consensus.cc:2229] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:57.678678 8541 pending_rounds.cc:62] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Trying to abort 2 pending ops.
I20260927 22:47:57.678845 8540 ts_tablet_manager.cc:1905] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.678813 8541 pending_rounds.cc:68] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334089841291632640 op_type: WRITE_OP write_request { tablet_id: "8b2a979bdf2c4d9a8f668cc2ccbfa04f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\311\274\346\325\006\"8\n\ntest-admin\022*\n 6d23a3b0830f44a48d29ff03d53b5db4\020\001\030\001 \001(\001" signature: "<redacted>""\2545\347on\247_5o&\007\3106\315\220\357\016\300\351\312}\013\233\237\202\3432\273\314\335\226\252n\035\344\227\2709\310\263\001\262\254\366\316w\344B\360\360\232\315I\001\333\327\213\n\234\254\327\325c\254" signing_key_seq_num: 0 } } request_id { client_id: "635c737da9aa43f8821f05b260915f8f" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 22:47:57.679526 8541 pending_rounds.cc:68] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334089841176977408 op_type: NO_OP noop_request { }
I20260927 22:47:57.679677 8541 raft_consensus.cc:2866] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 22:47:57.679795 8541 raft_consensus.cc:2258] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.680222 8099 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55262
I20260927 22:47:57.680469 8539 tablet_replica.cc:330] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: stopping tablet replica
I20260927 22:47:57.680641 8539 raft_consensus.cc:2229] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:57.680840 8539 raft_consensus.cc:2258] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.681806 8539 ts_tablet_manager.cc:1905] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.682740 7964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:47:57.683331 7964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:47:57.685127 8541 ts_tablet_manager.cc:1905] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.686282 8540 ts_tablet_manager.cc:1918] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:57.686427 8540 log.cc:1196] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/8b2a979bdf2c4d9a8f668cc2ccbfa04f
I20260927 22:47:57.686798 8540 ts_tablet_manager.cc:1939] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 6da16e51fc9347849f49280001f80f1b: Deleting consensus metadata
I20260927 22:47:57.687789 7948 catalog_manager.cc:4841] TS 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975): tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f (table test-table [id=6d23a3b0830f44a48d29ff03d53b5db4]) successfully deleted
I20260927 22:47:57.689141 8539 ts_tablet_manager.cc:1918] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:57.689280 8539 log.cc:1196] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/8b2a979bdf2c4d9a8f668cc2ccbfa04f
I20260927 22:47:57.689627 8539 ts_tablet_manager.cc:1939] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting consensus metadata
I20260927 22:47:57.690537 7951 catalog_manager.cc:4841] TS 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223): tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f (table test-table [id=6d23a3b0830f44a48d29ff03d53b5db4]) successfully deleted
I20260927 22:47:57.692497 8541 ts_tablet_manager.cc:1918] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:47:57.692633 8541 log.cc:1196] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/8b2a979bdf2c4d9a8f668cc2ccbfa04f
I20260927 22:47:57.692965 8541 ts_tablet_manager.cc:1939] T 8b2a979bdf2c4d9a8f668cc2ccbfa04f P 559c805f26eb4d23a7aa119364e42de4: Deleting consensus metadata
I20260927 22:47:57.693992 7949 catalog_manager.cc:4841] TS 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217): tablet 8b2a979bdf2c4d9a8f668cc2ccbfa04f (table test-table [id=6d23a3b0830f44a48d29ff03d53b5db4]) successfully deleted
I20260927 22:47:57.694427 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffaa035471b4f7da2a7f58563fecb80 (DEFAULT_TABLE table=test-table [id=e608fc3f7258420ca2ebe6615357921a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:57.694448 8099 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffaa035471b4f7da2a7f58563fecb80 (DEFAULT_TABLE table=test-table [id=e608fc3f7258420ca2ebe6615357921a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:57.694757 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffaa035471b4f7da2a7f58563fecb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:57.694773 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffaa035471b4f7da2a7f58563fecb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:57.698967 8556 tablet_bootstrap.cc:492] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: Bootstrap starting.
I20260927 22:47:57.699322 8377 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffaa035471b4f7da2a7f58563fecb80 (DEFAULT_TABLE table=test-table [id=e608fc3f7258420ca2ebe6615357921a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:47:57.699697 8377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffaa035471b4f7da2a7f58563fecb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:47:57.700470 8556 tablet_bootstrap.cc:654] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:57.700623 8555 tablet_bootstrap.cc:492] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: Bootstrap starting.
I20260927 22:47:57.702003 8555 tablet_bootstrap.cc:654] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:57.706696 8557 tablet_bootstrap.cc:492] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: Bootstrap starting.
I20260927 22:47:57.708400 8557 tablet_bootstrap.cc:654] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: Neither blocks nor log segments found. Creating new log.
I20260927 22:47:57.712352 8556 tablet_bootstrap.cc:492] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: No bootstrap required, opened a new log
I20260927 22:47:57.712527 8556 ts_tablet_manager.cc:1397] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260927 22:47:57.713403 8555 tablet_bootstrap.cc:492] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: No bootstrap required, opened a new log
I20260927 22:47:57.713320 8556 raft_consensus.cc:348] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.713573 8555 ts_tablet_manager.cc:1397] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260927 22:47:57.713579 8556 raft_consensus.cc:374] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:57.713649 8556 raft_consensus.cc:729] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9577120fc7174b6d8f9169bc9a85c1e1, State: Initialized, Role: FOLLOWER
I20260927 22:47:57.713953 8556 consensus_queue.cc:260] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.714511 8556 ts_tablet_manager.cc:1428] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:47:57.714936 8555 raft_consensus.cc:348] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.715160 8555 raft_consensus.cc:374] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:57.715248 8555 raft_consensus.cc:729] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da16e51fc9347849f49280001f80f1b, State: Initialized, Role: FOLLOWER
I20260927 22:47:57.715620 8557 tablet_bootstrap.cc:492] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: No bootstrap required, opened a new log
I20260927 22:47:57.715519 8555 consensus_queue.cc:260] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.715916 8557 ts_tablet_manager.cc:1397] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:47:57.716506 8555 ts_tablet_manager.cc:1428] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:47:57.716548 8557 raft_consensus.cc:348] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.716775 8557 raft_consensus.cc:374] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:47:57.716871 8557 raft_consensus.cc:729] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c805f26eb4d23a7aa119364e42de4, State: Initialized, Role: FOLLOWER
I20260927 22:47:57.717092 8557 consensus_queue.cc:260] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.717628 8557 ts_tablet_manager.cc:1428] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:47:57.719152 8480 raft_consensus.cc:482] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:47:57.719319 8480 raft_consensus.cc:504] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.721029 8480 leader_election.cc:288] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:57.728467 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.721695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53611 (local address 127.0.97.3:44217)
0927 22:47:57.721847 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.721853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.722024 (+ 171us) server_negotiation.cc:407] Connection header received
0927 22:47:57.722492 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.722495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.722542 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.722606 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:57.723167 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.724141 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.726332 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.727591 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.727690 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.727772 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.728094 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.728216 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:47:57.728543 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.721360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44217 (local address 127.0.97.1:53611)
0927 22:47:57.721960 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:57.721973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:57.722090 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:57.722643 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:57.722647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:57.722667 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:57.723009 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.723018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.724246 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.724251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.726221 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.726231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.727747 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.727753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.727936 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.727971 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:47:57.728047 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 22:47:57.728165 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":25,"spinlock_wait_cycles":6784,"thread_start_us":503,"threads_started":1}
I20260927 22:47:57.728602 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.721226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52413 (local address 127.0.97.2:44975)
0927 22:47:57.721367 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.721371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.722001 (+ 630us) server_negotiation.cc:407] Connection header received
0927 22:47:57.722229 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.722233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.722286 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.722369 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:47:57.722777 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.723872 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.726259 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.727510 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.727624 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.727713 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.728240 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.728402 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 22:47:57.729164 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffaa035471b4f7da2a7f58563fecb80" candidate_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4" is_pre_election: true
I20260927 22:47:57.729377 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.721294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44975 (local address 127.0.97.1:52413)
0927 22:47:57.721902 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:57.721925 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:57.722089 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:57.722378 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:57.722381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:57.722398 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:47:57.722648 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.722657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.723996 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.723999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.726111 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.726129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.727861 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.727867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.728032 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.728101 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:47:57.728906 (+ 805us) client_negotiation.cc:240] Negotiation successful
0927 22:47:57.729025 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":17,"spinlock_wait_cycles":1792,"thread_start_us":315,"threads_started":1}
I20260927 22:47:57.729386 8398 raft_consensus.cc:2454] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9577120fc7174b6d8f9169bc9a85c1e1 in term 0.
I20260927 22:47:57.729842 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffaa035471b4f7da2a7f58563fecb80" candidate_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da16e51fc9347849f49280001f80f1b" is_pre_election: true
I20260927 22:47:57.729972 8052 leader_election.cc:302] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 559c805f26eb4d23a7aa119364e42de4, 9577120fc7174b6d8f9169bc9a85c1e1; no voters:
I20260927 22:47:57.730060 8263 raft_consensus.cc:2454] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9577120fc7174b6d8f9169bc9a85c1e1 in term 0.
I20260927 22:47:57.730340 8480 raft_consensus.cc:2781] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:47:57.730464 8480 raft_consensus.cc:482] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:47:57.730556 8480 raft_consensus.cc:3037] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.732553 8480 raft_consensus.cc:504] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.733003 8480 leader_election.cc:288] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [CANDIDATE]: Term 1 election: Requested vote from peers 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975), 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217)
I20260927 22:47:57.733345 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffaa035471b4f7da2a7f58563fecb80" candidate_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da16e51fc9347849f49280001f80f1b"
I20260927 22:47:57.733461 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffaa035471b4f7da2a7f58563fecb80" candidate_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c805f26eb4d23a7aa119364e42de4"
I20260927 22:47:57.733530 8263 raft_consensus.cc:3037] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.733603 8398 raft_consensus.cc:3037] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:47:57.735389 8263 raft_consensus.cc:2454] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9577120fc7174b6d8f9169bc9a85c1e1 in term 1.
I20260927 22:47:57.735517 8398 raft_consensus.cc:2454] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9577120fc7174b6d8f9169bc9a85c1e1 in term 1.
I20260927 22:47:57.735785 8051 leader_election.cc:302] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6da16e51fc9347849f49280001f80f1b, 9577120fc7174b6d8f9169bc9a85c1e1; no voters:
I20260927 22:47:57.736053 8480 raft_consensus.cc:2781] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:47:57.736390 8480 raft_consensus.cc:686] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 LEADER]: Becoming Leader. State: Replica: 9577120fc7174b6d8f9169bc9a85c1e1, State: Running, Role: LEADER
I20260927 22:47:57.736696 8480 consensus_queue.cc:237] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } }
I20260927 22:47:57.739665 7963 catalog_manager.cc:5495] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9577120fc7174b6d8f9169bc9a85c1e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 } health_report { overall_health: UNKNOWN } } }
I20260927 22:47:57.764825 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.759738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.0.97.1:40223)
0927 22:47:57.760239 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.760243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.760261 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:47:57.760334 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.760337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.760407 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.760540 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:57.760863 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.761606 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.762798 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.763202 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.763288 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.763708 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.764290 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.764395 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.764458 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.764514 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":299,"threads_started":1}
I20260927 22:47:57.778046 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.773537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38109 (local address 127.0.97.3:44217)
0927 22:47:57.773678 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.773683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.774195 (+ 512us) server_negotiation.cc:407] Connection header received
0927 22:47:57.774311 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.774315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.774377 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.774448 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:57.774804 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.775500 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.776706 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.777109 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.777199 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.777544 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.777640 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.777703 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.777788 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.777843 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:47:57.778167 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.773521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44217 (local address 127.0.97.1:38109)
0927 22:47:57.774099 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:57.774117 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:57.774217 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:57.774459 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:57.774462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:57.774470 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:57.774699 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.774705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.775597 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.775601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.776600 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.776610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.777243 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.777248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.777367 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.777413 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.777708 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.777714 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:47:57.777764 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 22:47:57.777878 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":292,"threads_started":1}
I20260927 22:47:57.785384 8398 raft_consensus.cc:1261] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Refusing update from remote peer 9577120fc7174b6d8f9169bc9a85c1e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.785416 8263 raft_consensus.cc:1261] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Refusing update from remote peer 9577120fc7174b6d8f9169bc9a85c1e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:47:57.785959 8570 consensus_queue.cc:1035] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da16e51fc9347849f49280001f80f1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.786315 8480 consensus_queue.cc:1035] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "559c805f26eb4d23a7aa119364e42de4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:47:57.808027 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.803572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40223 (local address 127.0.97.3:54521)
0927 22:47:57.803734 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:47:57.803752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:47:57.803833 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:47:57.804154 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:47:57.804158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:47:57.804170 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:47:57.804363 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.804374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.805356 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.805360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.806380 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:47:57.806392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.807020 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.807025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:47:57.807178 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.807211 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.807543 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.807554 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:47:57.807628 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 22:47:57.807760 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 22:47:57.808049 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.803625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54521 (local address 127.0.97.1:40223)
0927 22:47:57.803776 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.803782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.803811 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:47:57.803926 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.803931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.804039 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.804129 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:47:57.804492 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.805237 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.806505 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.806888 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.807004 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.807340 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:47:57.807459 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:47:57.807540 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.807664 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.807733 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:47:57.833631 7963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:57.833912 7963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:47:57.837234 7963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7add240533804303979ff30267a2699d: Sending DeleteTablet for 3 replicas of tablet 3ffaa035471b4f7da2a7f58563fecb80
I20260927 22:47:57.837996 8099 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ffaa035471b4f7da2a7f58563fecb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55262
I20260927 22:47:57.838320 8539 tablet_replica.cc:330] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: stopping tablet replica
I20260927 22:47:57.838515 8539 raft_consensus.cc:2229] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:47:57.838671 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ffaa035471b4f7da2a7f58563fecb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44022
I20260927 22:47:57.838964 8377 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ffaa035471b4f7da2a7f58563fecb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:47:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39474
I20260927 22:47:57.839035 8539 raft_consensus.cc:2258] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.839229 8541 tablet_replica.cc:330] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: stopping tablet replica
I20260927 22:47:57.839340 8540 tablet_replica.cc:330] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: stopping tablet replica
I20260927 22:47:57.839388 8541 raft_consensus.cc:2229] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:57.839538 8540 raft_consensus.cc:2229] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:47:57.839617 8541 raft_consensus.cc:2258] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.839723 8540 raft_consensus.cc:2258] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:47:57.841183 8539 ts_tablet_manager.cc:1905] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.841641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 8035
I20260927 22:47:57.848263 8539 ts_tablet_manager.cc:1918] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:57.848521 8539 log.cc:1196] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/3ffaa035471b4f7da2a7f58563fecb80
I20260927 22:47:57.849027 8539 ts_tablet_manager.cc:1939] T 3ffaa035471b4f7da2a7f58563fecb80 P 9577120fc7174b6d8f9169bc9a85c1e1: Deleting consensus metadata
I20260927 22:47:57.849879 8541 ts_tablet_manager.cc:1905] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.850963 8540 ts_tablet_manager.cc:1905] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:47:57.856561 8541 ts_tablet_manager.cc:1918] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:57.856740 8541 log.cc:1196] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/3ffaa035471b4f7da2a7f58563fecb80
I20260927 22:47:57.857195 8541 ts_tablet_manager.cc:1939] T 3ffaa035471b4f7da2a7f58563fecb80 P 559c805f26eb4d23a7aa119364e42de4: Deleting consensus metadata
I20260927 22:47:57.857475 8540 ts_tablet_manager.cc:1918] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:47:57.857605 8540 log.cc:1196] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/3ffaa035471b4f7da2a7f58563fecb80
I20260927 22:47:57.857986 8540 ts_tablet_manager.cc:1939] T 3ffaa035471b4f7da2a7f58563fecb80 P 6da16e51fc9347849f49280001f80f1b: Deleting consensus metadata
I20260927 22:47:57.858517 7949 catalog_manager.cc:4841] TS 559c805f26eb4d23a7aa119364e42de4 (127.0.97.3:44217): tablet 3ffaa035471b4f7da2a7f58563fecb80 (table test-table [id=e608fc3f7258420ca2ebe6615357921a]) successfully deleted
I20260927 22:47:57.859004 7948 catalog_manager.cc:4841] TS 6da16e51fc9347849f49280001f80f1b (127.0.97.2:44975): tablet 3ffaa035471b4f7da2a7f58563fecb80 (table test-table [id=e608fc3f7258420ca2ebe6615357921a]) successfully deleted
I20260927 22:47:57.860184 7951 catalog_manager.cc:4841] TS 9577120fc7174b6d8f9169bc9a85c1e1 (127.0.97.1:40223): tablet 3ffaa035471b4f7da2a7f58563fecb80 (table test-table [id=e608fc3f7258420ca2ebe6615357921a]) successfully deleted
Sep 27 22:47:57 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 22:47:57.876937 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.843220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.0.97.1:40223)
0927 22:47:57.843401 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.843407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.860955 (+ 17548us) server_negotiation.cc:407] Connection header received
0927 22:47:57.861123 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.861128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.861226 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:57.861344 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:57.862793 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.863807 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.865603 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:57.866251 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:57.866348 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:57.870930 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:57.870947 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:57.870973 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:57.871013 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:57.874413 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:57.874988 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:57.874997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:57.875001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:57.875135 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:57.875490 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:57.875495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:57.875499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:57.875815 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:57.876163 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:57.876389 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 22:47:57.876588 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:47:57.877816 8165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:57.983142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 8035
I20260927 22:47:57.998797 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 8174
Sep 27 22:47:58 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 22:47:58.012836 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:57.999499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.0.97.2:44975)
0927 22:47:57.999676 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:57.999681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:57.999700 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:47:57.999887 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:57.999891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:57.999956 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:58.000042 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:58.001157 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:58.001900 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:58.003008 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:58.003631 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:58.003725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:58.007554 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:58.007580 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:58.007598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:58.007640 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:58.010389 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:58.011033 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:58.011040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:58.011043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:58.011166 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:58.011615 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:58.011619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:58.011622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:58.011978 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:58.012194 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:58.012410 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 22:47:58.012560 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:47:58.013441 8304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:58.130631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 8174
I20260927 22:47:58.147864 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 8313
Sep 27 22:47:58 dist-test-slave-kr25 krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 22:47:58.161397 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:47:58.148536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.0.97.3:44217)
0927 22:47:58.148704 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:47:58.148710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:47:58.148773 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:47:58.148951 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:47:58.148957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:47:58.149024 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:47:58.149115 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:47:58.150288 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:58.150988 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:58.152246 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:47:58.152648 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:47:58.152752 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:47:58.156640 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:47:58.156654 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:47:58.156670 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:47:58.156701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:47:58.159102 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:58.159804 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:58.159810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:58.159813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:58.159922 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:47:58.160188 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:47:58.160191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:47:58.160193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:47:58.160523 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:47:58.160744 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:47:58.160921 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 22:47:58.161081 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:47:58.161981 8443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:58.278520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 8313
I20260927 22:47:58.295174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 7932
I20260927 22:47:58.296355 8004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:47:58.406420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 7932
2026-09-27T22:47:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5180 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:47:58 dist-test-slave-kr25 krb5kdc[8587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:47:58 dist-test-slave-kr25 krb5kdc[8587](info): set up 2 sockets
Sep 27 22:47:58 dist-test-slave-kr25 krb5kdc[8587](info): commencing operation
krb5kdc: starting...
W20260927 22:47:59.408169 388 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:47:59 dist-test-slave-kr25 krb5kdc[8587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:47:59.606207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37357
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:47:59.907666 8600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:47:59.907990 8600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:47:59.908293 8600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:47:59.915452 8600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:47:59.915556 8600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:47:59.915627 8600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:47:59.915702 8600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:47:59.915766 8600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:47:59.925387 8600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:47:59.926628 8600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:47:59.927892 8600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:47:59.938452 8605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:59.938475 8606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:47:59.938634 8608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:47:59.939236 8600 server_base.cc:1034] running on GCE node
I20260927 22:47:59.940066 8600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:47:59.941476 8600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:47:59.942699 8600 hybrid_clock.cc:648] HybridClock initialized: now 1790549279942638 us; error 41 us; skew 500 ppm
I20260927 22:47:59.943058 8600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T22:48:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1606 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:48:00 dist-test-slave-kr25 krb5kdc[8614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:48:00 dist-test-slave-kr25 krb5kdc[8614](info): set up 2 sockets
Sep 27 22:48:00 dist-test-slave-kr25 krb5kdc[8614](info): commencing operation
krb5kdc: starting...
W20260927 22:48:01.020133 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:48:01 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:48:01.219338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46439
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:01.531783 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:01.532069 8627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:01.532197 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:01.539652 8627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:48:01.539747 8627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:01.539825 8627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:01.539904 8627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:48:01.539963 8627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:48:01.549635 8627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:01.550979 8627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:01.552546 8627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:01.562961 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:01.563756 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:01.564721 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:01.564950 8627 server_base.cc:1034] running on GCE node
I20260927 22:48:01.565611 8627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:01.566876 8627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:01.568068 8627 hybrid_clock.cc:648] HybridClock initialized: now 1790549281568028 us; error 28 us; skew 500 ppm
I20260927 22:48:01.568363 8627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:01 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:01.574877 8627 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:48:01.576263 8627 webserver.cc:460] Webserver started at http://127.0.97.62:34821/ using document root <none> and password file <none>
I20260927 22:48:01.576910 8627 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:01.577015 8627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:01.577286 8627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:01.580097 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "75dda5a6ba484a9380e2ec4fa169ed2f"
format_stamp: "Formatted at 2026-09-27 22:48:01 on dist-test-slave-kr25"
server_key: "92dd7acbb6f227664bf038ae1ff47017"
server_key_iv: "5d88dfabda7d34088d833e4260d4866f"
server_key_version: "encryptionkey@0"
I20260927 22:48:01.580796 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "75dda5a6ba484a9380e2ec4fa169ed2f"
format_stamp: "Formatted at 2026-09-27 22:48:01 on dist-test-slave-kr25"
server_key: "92dd7acbb6f227664bf038ae1ff47017"
server_key_iv: "5d88dfabda7d34088d833e4260d4866f"
server_key_version: "encryptionkey@0"
I20260927 22:48:01.585538 8627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:48:01.588836 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:01.590126 8627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:48:01.590284 8627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "75dda5a6ba484a9380e2ec4fa169ed2f"
format_stamp: "Formatted at 2026-09-27 22:48:01 on dist-test-slave-kr25"
server_key: "92dd7acbb6f227664bf038ae1ff47017"
server_key_iv: "5d88dfabda7d34088d833e4260d4866f"
server_key_version: "encryptionkey@0"
I20260927 22:48:01.590458 8627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:01.608096 8627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:01.611852 8627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:01.612103 8627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:01.628646 8627 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46439
I20260927 22:48:01.628676 8693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46439 every 8 connection(s)
I20260927 22:48:01.629908 8627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:48:01.633529 8694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:48:01.636641 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 8627
I20260927 22:48:01.636831 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:48:01.637625 388 external_mini_cluster.cc:1428] Setting key b8f750e19cd80d4c61da128435de5a3d
I20260927 22:48:01.645678 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Bootstrap starting.
I20260927 22:48:01.649209 8694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Neither blocks nor log segments found. Creating new log.
I20260927 22:48:01.650404 8694 log.cc:824] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:48:01 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:01.653460 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: No bootstrap required, opened a new log
I20260927 22:48:01.659999 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:01.639612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.0.97.62:46439)
0927 22:48:01.640298 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:01.640309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:01.640351 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:48:01.641907 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:01.641929 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:01.642323 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:01.642636 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:01.644301 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:01.645536 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:01.647130 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:01.647820 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:01.647969 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:01.652568 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:01.652576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:01.652590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:01.652624 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:01.655932 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:01.656673 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:01.656679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:01.656685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:01.656914 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:01.657288 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:01.657292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:01.657294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:01.657900 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:01.658123 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:01.658574 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 22:48:01.658939 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":311,"threads_started":1}
I20260927 22:48:01.661597 8694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:01.661914 8694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:48:01.662005 8694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75dda5a6ba484a9380e2ec4fa169ed2f, State: Initialized, Role: FOLLOWER
I20260927 22:48:01.662611 8694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:01.662828 8694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:48:01.662949 8694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:48:01.663091 8694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:48:01.665194 8694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:01.665644 8694 leader_election.cc:302] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75dda5a6ba484a9380e2ec4fa169ed2f; no voters:
I20260927 22:48:01.666141 8694 leader_election.cc:288] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:48:01.666232 8699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:48:01.666824 8699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 LEADER]: Becoming Leader. State: Replica: 75dda5a6ba484a9380e2ec4fa169ed2f, State: Running, Role: LEADER
I20260927 22:48:01.667297 8699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:01.667930 8694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:48:01.670775 8701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75dda5a6ba484a9380e2ec4fa169ed2f. Latest consensus state: current_term: 1 leader_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } } }
I20260927 22:48:01.670748 8700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } } }
I20260927 22:48:01.671193 8701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:01.671259 8700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:01.672494 8710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:48:01.677357 8710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:48:01.685107 8710 catalog_manager.cc:1347] Generated new cluster ID: 51087a151b424ef587fc2ae147e43e75
I20260927 22:48:01.685206 8710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:48:01.718137 8710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:48:01.719463 8710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:48:01.728592 8710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Generated new TSK 0
I20260927 22:48:01.729344 8710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:48:01.788417 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:02.140452 8720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:02.140720 8720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:02.140844 8720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:02.147889 8720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:02.148013 8720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:02.148175 8720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:48:02.157820 8720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:02.159101 8720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:02.160410 8720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:02.172823 8725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:02.172777 8728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:02.172749 8726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:02.173786 8720 server_base.cc:1034] running on GCE node
I20260927 22:48:02.174337 8720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:02.175195 8720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:02.176383 8720 hybrid_clock.cc:648] HybridClock initialized: now 1790549282176330 us; error 47 us; skew 500 ppm
I20260927 22:48:02.176658 8720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:02 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:02.179569 8720 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:48:02.180825 8720 webserver.cc:460] Webserver started at http://127.0.97.1:40603/ using document root <none> and password file <none>
I20260927 22:48:02.181442 8720 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:02.181545 8720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:02.181823 8720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:02.184615 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "db882807c74a490a91961014eca2ac6b"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "ca9511b80594c9ebc29ff01873b3f428"
server_key_iv: "b3744e979c7b259e32b3caf407a839be"
server_key_version: "encryptionkey@0"
I20260927 22:48:02.185328 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "db882807c74a490a91961014eca2ac6b"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "ca9511b80594c9ebc29ff01873b3f428"
server_key_iv: "b3744e979c7b259e32b3caf407a839be"
server_key_version: "encryptionkey@0"
I20260927 22:48:02.190145 8720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:48:02.193431 8735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:02.194720 8720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:48:02.194855 8720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "db882807c74a490a91961014eca2ac6b"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "ca9511b80594c9ebc29ff01873b3f428"
server_key_iv: "b3744e979c7b259e32b3caf407a839be"
server_key_version: "encryptionkey@0"
I20260927 22:48:02.195039 8720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:02.211678 8720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:02.215723 8720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:02.215976 8720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:02.216843 8720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:02.218086 8720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:02.218155 8720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:02.218269 8720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:02.218317 8720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:02.246039 8720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41815
I20260927 22:48:02.246086 8847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41815 every 8 connection(s)
I20260927 22:48:02.247202 8720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:48:02.250111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 8720
I20260927 22:48:02.250274 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:48:02.251086 388 external_mini_cluster.cc:1428] Setting key e0bf3b922fbee3c1e8b5da325999de02
Sep 27 22:48:02 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:02.267728 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:02.249499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44429 (local address 127.0.97.62:46439)
0927 22:48:02.249908 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:02.249915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:02.251086 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 22:48:02.253362 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:02.253365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:02.253436 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:02.253532 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:02.255387 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.256133 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.258082 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.258455 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.258538 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:02.262022 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:02.262028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:02.262032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:02.262067 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:02.264951 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:02.265585 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:02.265590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:02.265594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:02.265716 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:02.266051 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:02.266056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:02.266059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:02.266377 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:02.266502 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:02.267139 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 22:48:02.267341 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":190,"threads_started":1}
I20260927 22:48:02.268671 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:02.250060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:44429)
0927 22:48:02.250901 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:02.250949 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:02.253077 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:02.253837 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:02.253847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:02.254495 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:02.255215 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:02.255236 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.256275 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.256280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:02.257965 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:02.257977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.258643 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.258649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:02.258907 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:02.259554 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:02.259584 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:02.261822 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:02.265082 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:02.265088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:02.265100 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:02.265489 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:02.265802 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:02.265805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:02.265808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:02.265965 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:02.266469 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:02.266475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:02.266766 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 22:48:02.267028 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 22:48:02.267387 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":293,"threads_started":1}
I20260927 22:48:02.270099 8848 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46439
I20260927 22:48:02.270434 8848 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:02.271286 8848 heartbeater.cc:511] Master 127.0.97.62:46439 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:48:02.273545 8659 ts_manager.cc:194] Registered new tserver with Master: db882807c74a490a91961014eca2ac6b (127.0.97.1:41815)
I20260927 22:48:02.275230 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:48:02.298503 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:02.600651 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:02.600930 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:02.601068 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:02.608017 8854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:02.608132 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:02.608278 8854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:48:02.617499 8854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:02.618722 8854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:02.619992 8854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:02.631919 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:02.631943 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:02.632481 8862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:02.633085 8854 server_base.cc:1034] running on GCE node
I20260927 22:48:02.633759 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:02.634833 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:02.636075 8854 hybrid_clock.cc:648] HybridClock initialized: now 1790549282636009 us; error 53 us; skew 500 ppm
I20260927 22:48:02.636440 8854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:02 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:02.640120 8854 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:48:02.641716 8854 webserver.cc:460] Webserver started at http://127.0.97.2:34395/ using document root <none> and password file <none>
I20260927 22:48:02.642529 8854 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:02.642655 8854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:02.642992 8854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:02.646636 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "0c231e4c6f864505a795a748c51a4032"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "8ec704cbe48c9a983dc2d412b4ccb8ae"
server_key_iv: "28261f7688c8ced9d33cdb2e7f2b8a5f"
server_key_version: "encryptionkey@0"
I20260927 22:48:02.647544 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0c231e4c6f864505a795a748c51a4032"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "8ec704cbe48c9a983dc2d412b4ccb8ae"
server_key_iv: "28261f7688c8ced9d33cdb2e7f2b8a5f"
server_key_version: "encryptionkey@0"
I20260927 22:48:02.652283 8854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:48:02.655321 8869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:02.656555 8854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:48:02.656677 8854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "0c231e4c6f864505a795a748c51a4032"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "8ec704cbe48c9a983dc2d412b4ccb8ae"
server_key_iv: "28261f7688c8ced9d33cdb2e7f2b8a5f"
server_key_version: "encryptionkey@0"
I20260927 22:48:02.656824 8854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:02.670933 8854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:02.674818 8854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:02.675077 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:02.676033 8854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:02.677265 8854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:02.677335 8854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:02.677445 8854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:02.677492 8854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:02.705303 8854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40109
I20260927 22:48:02.705356 8981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40109 every 8 connection(s)
I20260927 22:48:02.706480 8854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:48:02.715374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 8854
I20260927 22:48:02.715564 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:48:02.716081 388 external_mini_cluster.cc:1428] Setting key a4ed2ee1cea6b0b217e8fe389ee69284
Sep 27 22:48:02 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:02.724881 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:02.708794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35779 (local address 127.0.97.62:46439)
0927 22:48:02.708979 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:02.708985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:02.710094 (+ 1109us) server_negotiation.cc:407] Connection header received
0927 22:48:02.711797 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:02.711802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:02.711865 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:02.711953 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:02.713262 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.713962 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.715413 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.716094 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.716224 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:02.719660 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:02.719665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:02.719669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:02.719698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:02.722004 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:02.722761 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:02.722765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:02.722767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:02.722884 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:02.723277 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:02.723280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:02.723282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:02.723557 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:02.723661 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:02.724319 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 22:48:02.724546 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:48:02.725616 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:02.709266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:35779)
0927 22:48:02.709938 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:02.709985 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:02.711562 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:02.712132 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:02.712140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:02.712602 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:02.713136 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:02.713152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.714078 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.714081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:02.715268 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:02.715279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:02.716317 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:02.716323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:02.716508 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:02.717138 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:02.717159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:02.719408 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:02.722121 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:02.722129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:02.722146 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:02.722667 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:02.722970 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:02.722974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:02.722977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:02.723186 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:02.723657 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:02.723664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:02.724004 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 22:48:02.724218 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:48:02.724586 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260927 22:48:02.726939 8982 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46439
I20260927 22:48:02.727237 8982 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:02.728073 8982 heartbeater.cc:511] Master 127.0.97.62:46439 requested a full tablet report, sending...
I20260927 22:48:02.729548 8659 ts_manager.cc:194] Registered new tserver with Master: 0c231e4c6f864505a795a748c51a4032 (127.0.97.2:40109)
I20260927 22:48:02.730298 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35779
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:48:02.760334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:03.062316 8987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:03.062579 8987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:03.062696 8987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:03.069849 8987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:03.069963 8987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:03.070101 8987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:48:03.079751 8987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:03.080963 8987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:03.082198 8987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:03.094211 8992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:03.094296 8995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:03.094249 8993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:03.095244 8987 server_base.cc:1034] running on GCE node
I20260927 22:48:03.095892 8987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:03.096736 8987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:03.097985 8987 hybrid_clock.cc:648] HybridClock initialized: now 1790549283097935 us; error 39 us; skew 500 ppm
I20260927 22:48:03.098273 8987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:03 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:03.101245 8987 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:48:03.102602 8987 webserver.cc:460] Webserver started at http://127.0.97.3:43991/ using document root <none> and password file <none>
I20260927 22:48:03.103212 8987 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:03.103310 8987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:03.103606 8987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:03.106379 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "82fc2b015c6840408727ae21c81d6922"
format_stamp: "Formatted at 2026-09-27 22:48:03 on dist-test-slave-kr25"
server_key: "dee5e3b1282f79dfbeaaaaf2a945cfa0"
server_key_iv: "c41e3bed613c7c928a1f355c2b9d2d01"
server_key_version: "encryptionkey@0"
I20260927 22:48:03.107064 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "82fc2b015c6840408727ae21c81d6922"
format_stamp: "Formatted at 2026-09-27 22:48:03 on dist-test-slave-kr25"
server_key: "dee5e3b1282f79dfbeaaaaf2a945cfa0"
server_key_iv: "c41e3bed613c7c928a1f355c2b9d2d01"
server_key_version: "encryptionkey@0"
I20260927 22:48:03.111788 8987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 22:48:03.115072 9002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:03.116494 8987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:48:03.116636 8987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "82fc2b015c6840408727ae21c81d6922"
format_stamp: "Formatted at 2026-09-27 22:48:03 on dist-test-slave-kr25"
server_key: "dee5e3b1282f79dfbeaaaaf2a945cfa0"
server_key_iv: "c41e3bed613c7c928a1f355c2b9d2d01"
server_key_version: "encryptionkey@0"
I20260927 22:48:03.116811 8987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:03.151365 8987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:03.155774 8987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:03.156039 8987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:03.157017 8987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:03.158316 8987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:03.158387 8987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:03.158509 8987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:03.158558 8987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:03.187398 8987 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46439
I20260927 22:48:03.187464 9114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46439 every 8 connection(s)
I20260927 22:48:03.188620 8987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:48:03.197567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 8987
I20260927 22:48:03.197715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:48:03.198127 388 external_mini_cluster.cc:1428] Setting key f4cfc99b020553f5948080d8836fe58a
Sep 27 22:48:03 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:03.207011 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:03.191004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42447 (local address 127.0.97.62:46439)
0927 22:48:03.191175 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:03.191180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:03.192389 (+ 1209us) server_negotiation.cc:407] Connection header received
0927 22:48:03.194188 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:03.194194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:03.194271 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:03.194356 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:03.195894 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.196634 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.198238 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.198661 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.198856 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:03.202441 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:03.202446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:03.202449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:03.202480 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:03.204704 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.205305 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.205308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.205310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.205420 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.205734 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.205736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.205738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.205963 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:03.206060 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:03.206631 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 22:48:03.206779 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:48:03.207949 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:03.191539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:42447)
0927 22:48:03.192242 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:03.192279 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:03.193932 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:03.194606 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:03.194615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:03.195103 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:03.195731 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:03.195749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.196789 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.196792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:03.198107 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:03.198119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.198952 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.198960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:03.199211 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:03.199916 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:03.199953 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:03.202241 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:03.204819 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:03.204825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:03.204837 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:03.205218 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:03.205491 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:03.205493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:03.205496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:03.205658 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:03.206048 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:03.206053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:03.206340 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 22:48:03.206549 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 22:48:03.206898 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":258,"threads_started":1}
I20260927 22:48:03.209225 9115 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46439
I20260927 22:48:03.209467 9115 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:03.210091 9115 heartbeater.cc:511] Master 127.0.97.62:46439 requested a full tablet report, sending...
I20260927 22:48:03.211395 8659 ts_manager.cc:194] Registered new tserver with Master: 82fc2b015c6840408727ae21c81d6922 (127.0.97.3:46439)
I20260927 22:48:03.212431 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42447
I20260927 22:48:03.213217 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 22:48:03.278187 8848 heartbeater.cc:503] Master 127.0.97.62:46439 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:48:03.330634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 8720
Sep 27 22:48:03 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:48:03.345508 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:03.331771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.0.97.1:41815)
0927 22:48:03.332405 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:03.332412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:03.332438 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:48:03.332496 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:03.332502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:03.332706 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:03.332832 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:03.334018 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.334997 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.336200 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.336922 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.337049 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:03.340558 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:03.340568 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:03.340582 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:03.340611 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:03.343263 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.343890 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.343897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.343899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.344017 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.344277 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.344280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.344282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.344581 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:03.344789 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:03.344984 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:48:03.345164 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":338,"threads_started":1}
I20260927 22:48:03.346642 8843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:03.426580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 8720
I20260927 22:48:03.439419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 8854
Sep 27 22:48:03 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:48:03.453766 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:03.440266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.0.97.2:40109)
0927 22:48:03.440727 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:03.440734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:03.440758 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:48:03.440869 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:03.440875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:03.441073 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:03.441190 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:03.442315 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.443367 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.444635 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.445171 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.445268 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:03.448907 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:03.448918 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:03.448933 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:03.448964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:03.451397 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.452034 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.452041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.452043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.452166 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.452474 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.452477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.452479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.452802 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:03.453023 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:03.453227 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:48:03.453423 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260927 22:48:03.454967 8977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:03.534701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 8854
I20260927 22:48:03.546836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 8987
Sep 27 22:48:03 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:48:03.562633 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:03.547690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.0.97.3:46439)
0927 22:48:03.548256 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:03.548263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:03.548290 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:48:03.548346 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:03.548351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:03.548552 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:03.548695 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:03.549909 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.550916 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.552237 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:03.552790 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:03.552901 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:03.556911 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:03.556922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:03.556936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:03.556968 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:03.559960 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.560724 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.560734 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.560736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.560875 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:03.561282 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:03.561287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:03.561291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:03.561610 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:03.561841 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:03.562119 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:48:03.562358 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":256,"threads_started":1}
I20260927 22:48:03.563925 9110 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 22:48:03.633018 8690 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.62:46439
I20260927 22:48:03.647897 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 8987
I20260927 22:48:03.659934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 8627
I20260927 22:48:03.660991 8689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:03.759200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 8627
I20260927 22:48:03.775735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46439
--webserver_interface=127.0.97.62
--webserver_port=34821
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:04.080200 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:04.080471 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:04.080592 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:04.087674 9132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:48:04.087786 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:04.087862 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:04.087944 9132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:48:04.088004 9132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:48:04.097647 9132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:04.098909 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:04.100225 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:04.110870 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:04.110912 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:04.110983 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:04.111821 9132 server_base.cc:1034] running on GCE node
I20260927 22:48:04.112540 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:04.113857 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:04.115079 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790549284115035 us; error 33 us; skew 500 ppm
I20260927 22:48:04.115389 9132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:04.118330 9132 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:48:04.119843 9132 webserver.cc:460] Webserver started at http://127.0.97.62:34821/ using document root <none> and password file <none>
I20260927 22:48:04.120489 9132 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:04.120615 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:04.125824 9132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:48:04.128537 9147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:04.129895 9132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:48:04.130120 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "75dda5a6ba484a9380e2ec4fa169ed2f"
format_stamp: "Formatted at 2026-09-27 22:48:01 on dist-test-slave-kr25"
server_key: "92dd7acbb6f227664bf038ae1ff47017"
server_key_iv: "5d88dfabda7d34088d833e4260d4866f"
server_key_version: "encryptionkey@0"
I20260927 22:48:04.131062 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:04.155202 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:04.159178 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:04.159413 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:04.175674 9132 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46439
I20260927 22:48:04.175680 9198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46439 every 8 connection(s)
I20260927 22:48:04.177105 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:48:04.182756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 9132
I20260927 22:48:04.183714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41815
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46439
--webserver_port=40603
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:48:04.184000 9199 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:48:04.188390 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Bootstrap starting.
I20260927 22:48:04.207079 9199 log.cc:824] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:48:04.215535 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:48:04.215983 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Bootstrap complete.
I20260927 22:48:04.224452 9199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:04.225237 9199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75dda5a6ba484a9380e2ec4fa169ed2f, State: Initialized, Role: FOLLOWER
I20260927 22:48:04.225782 9199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:04.225993 9199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:48:04.226089 9199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:48:04.226239 9199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:48:04.229857 9199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:04.230376 9199 leader_election.cc:302] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75dda5a6ba484a9380e2ec4fa169ed2f; no voters:
I20260927 22:48:04.230989 9199 leader_election.cc:288] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:48:04.231123 9203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:48:04.232224 9203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [term 2 LEADER]: Becoming Leader. State: Replica: 75dda5a6ba484a9380e2ec4fa169ed2f, State: Running, Role: LEADER
I20260927 22:48:04.232790 9203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260927 22:48:04.233134 9199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:48:04.235397 9204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } } }
I20260927 22:48:04.235921 9204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:04.235714 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75dda5a6ba484a9380e2ec4fa169ed2f. Latest consensus state: current_term: 2 leader_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dda5a6ba484a9380e2ec4fa169ed2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } } }
I20260927 22:48:04.236686 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:04.248610 9210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:48:04.254496 9210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:48:04.255995 9210 catalog_manager.cc:1259] Loaded cluster ID: 51087a151b424ef587fc2ae147e43e75
I20260927 22:48:04.256150 9210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:48:04.260165 9210 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:48:04.261638 9210 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75dda5a6ba484a9380e2ec4fa169ed2f: Loaded TSK: 0
I20260927 22:48:04.263227 9210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:48:04.266155 9220 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 22:48:04.538203 9201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:04.538480 9201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:04.538610 9201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:04.545814 9201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:04.545941 9201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:04.546110 9201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:48:04.555703 9201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40603
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:04.557157 9201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:04.558559 9201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:04.573549 9225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:04.573570 9226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:04.575665 9228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:04.576830 9201 server_base.cc:1034] running on GCE node
I20260927 22:48:04.577395 9201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:04.578302 9201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:04.579516 9201 hybrid_clock.cc:648] HybridClock initialized: now 1790549284579467 us; error 43 us; skew 500 ppm
I20260927 22:48:04.579802 9201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549284, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:04.582602 9201 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:48:04.584050 9201 webserver.cc:460] Webserver started at http://127.0.97.1:40603/ using document root <none> and password file <none>
I20260927 22:48:04.584685 9201 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:04.584829 9201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:04.589973 9201 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:48:04.592617 9235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:04.594012 9201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:48:04.594147 9201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "db882807c74a490a91961014eca2ac6b"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "ca9511b80594c9ebc29ff01873b3f428"
server_key_iv: "b3744e979c7b259e32b3caf407a839be"
server_key_version: "encryptionkey@0"
I20260927 22:48:04.595068 9201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:04.606679 9201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:04.610520 9201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:04.610749 9201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:04.611724 9201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:04.613008 9201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:04.613111 9201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:04.613209 9201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:04.613273 9201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:04.644366 9201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41815
I20260927 22:48:04.644425 9347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41815 every 8 connection(s)
I20260927 22:48:04.645721 9201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:48:04.646131 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 9201
I20260927 22:48:04.646796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40109
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46439
--webserver_port=34395
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:04.666180 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:04.648800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:42385)
0927 22:48:04.649746 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:04.649809 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:04.651899 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:04.653405 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:04.653416 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:04.654106 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:04.654817 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:04.654840 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.656427 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.656432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:04.658205 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:04.658218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.659180 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.659187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:04.659428 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:04.660116 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:04.660143 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:04.664790 (+ 4647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":491,"threads_started":1}
W20260927 22:48:04.667201 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:04.648735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42385 (local address 127.0.97.62:46439)
0927 22:48:04.649529 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:04.649578 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:04.649933 (+ 355us) server_negotiation.cc:407] Connection header received
0927 22:48:04.652487 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:04.652504 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:04.652894 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:04.653235 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:04.654993 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.656276 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.658347 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.658977 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.659102 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:04.665439 (+ 6337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42385: BlockingRecv error: recv got EOF from 127.0.97.1:42385 (error 108)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":236,"threads_started":1}
W20260927 22:48:04.667744 9348 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46439 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46439: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:04.676789 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:04.668462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:37389)
0927 22:48:04.668695 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:04.668713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:04.668841 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:04.669278 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:04.669283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:04.669693 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:04.669886 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:04.669898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.671307 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.671321 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:04.672556 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:04.672568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.673219 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.673225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:04.673404 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:04.673960 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:04.673978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:04.676534 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260927 22:48:04.676963 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:04.668461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37389 (local address 127.0.97.62:46439)
0927 22:48:04.668777 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:04.668793 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:04.668814 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:48:04.668970 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:04.668976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:04.669103 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:04.669200 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:04.670068 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.671174 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.672683 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.673087 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.673192 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:04.676746 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37389: BlockingRecv error: recv got EOF from 127.0.97.1:37389 (error 108)
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":164}
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:04.684520 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:04.677423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:38373)
0927 22:48:04.677558 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:04.677575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:04.677652 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:04.678060 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:04.678065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:04.678343 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:04.678483 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:04.678491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.679414 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.679419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:04.680325 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:04.680336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.680950 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.680956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:04.681112 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:04.681602 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:04.681618 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:04.684254 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 22:48:04.684726 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:04.677630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38373 (local address 127.0.97.62:46439)
0927 22:48:04.677805 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:04.677817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:04.677829 (+ 12us) server_negotiation.cc:407] Connection header received
0927 22:48:04.677870 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:04.677874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:04.677940 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:04.678020 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:04.678607 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.679288 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.680449 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:04.680830 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:04.680928 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:04.684425 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38373: BlockingRecv error: recv got EOF from 127.0.97.1:38373 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:48:04.684785 9348 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:48:04.955540 9352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:04.955818 9352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:04.955938 9352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:04.962921 9352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:04.963059 9352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:04.963217 9352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:48:04.972842 9352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34395
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:04.974130 9352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:04.975329 9352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:04.991729 9358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:04.991780 9357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:04.991760 9360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:04.992600 9352 server_base.cc:1034] running on GCE node
I20260927 22:48:04.993183 9352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:04.994086 9352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:04.995288 9352 hybrid_clock.cc:648] HybridClock initialized: now 1790549284995238 us; error 44 us; skew 500 ppm
I20260927 22:48:04.995623 9352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:04 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549284, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:04.998694 9352 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:48:05.000069 9352 webserver.cc:460] Webserver started at http://127.0.97.2:34395/ using document root <none> and password file <none>
I20260927 22:48:05.000702 9352 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:05.000818 9352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:05.005945 9352 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:48:05.008603 9367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:05.009915 9352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:48:05.010046 9352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "0c231e4c6f864505a795a748c51a4032"
format_stamp: "Formatted at 2026-09-27 22:48:02 on dist-test-slave-kr25"
server_key: "8ec704cbe48c9a983dc2d412b4ccb8ae"
server_key_iv: "28261f7688c8ced9d33cdb2e7f2b8a5f"
server_key_version: "encryptionkey@0"
I20260927 22:48:05.010937 9352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:05.037853 9352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:05.041994 9352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:05.042237 9352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:05.043190 9352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:05.044440 9352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:05.044528 9352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:05.044620 9352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:05.044682 9352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:05.073292 9352 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40109
I20260927 22:48:05.073362 9479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40109 every 8 connection(s)
I20260927 22:48:05.074658 9352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:48:05.079026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 9352
I20260927 22:48:05.079584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46439
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46439
--webserver_port=43991
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:48:05.089838 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:05.076985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38685 (local address 127.0.97.62:46439)
0927 22:48:05.077194 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.077200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.078340 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 22:48:05.080189 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.080195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.080271 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.080370 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.081738 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.082636 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.084125 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.084681 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.084793 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.089603 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38685: BlockingRecv error: recv got EOF from 127.0.97.2:38685 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:48:05.090341 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.077496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:38685)
0927 22:48:05.078183 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:05.078220 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:05.079930 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:05.080563 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:05.080572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:05.081053 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:05.081580 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.081597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.082767 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.082770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.083991 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.084016 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.084848 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.084853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.085044 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.085704 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:05.085730 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:05.089397 (+ 3667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":263,"threads_started":1}
W20260927 22:48:05.090974 9480 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46439 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46439: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:05.098428 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.091480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:48205)
0927 22:48:05.091607 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:05.091624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:05.091711 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:05.092002 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:05.092006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:05.092302 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:05.092431 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.092439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.093439 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.093443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.094300 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.094311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.094927 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.094933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.095082 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.095586 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:05.095602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:05.098167 (+ 2565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 22:48:05.098485 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:05.091579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48205 (local address 127.0.97.62:46439)
0927 22:48:05.091733 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.091738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.091755 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:48:05.091825 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.091830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.091889 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.091969 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.092554 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.093285 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.094422 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.094802 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.094907 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.098317 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48205: BlockingRecv error: recv got EOF from 127.0.97.2:48205 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:05.105662 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.099093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:44153)
0927 22:48:05.099220 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:05.099237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:05.099319 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:05.099613 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:05.099617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:05.099879 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:05.100004 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.100012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.100895 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.100900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.101755 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.101765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.102366 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.102371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.102521 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.102998 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:05.103013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:05.105390 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 22:48:05.105873 9480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:48:05.106078 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:05.099202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44153 (local address 127.0.97.62:46439)
0927 22:48:05.099361 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.099367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.099382 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:48:05.099423 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.099428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.099509 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.099583 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.100121 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.100780 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.101873 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.102248 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.102344 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.105581 (+ 3237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44153: BlockingRecv error: recv got EOF from 127.0.97.2:44153 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 22:48:05.391844 9483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:05.392134 9483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:05.392271 9483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:05.399567 9483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:05.399689 9483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:05.399853 9483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:48:05.409468 9483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43991
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:05.410720 9483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:05.411983 9483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:05.424029 9488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:05.424096 9491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:05.424050 9489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:05.424619 9483 server_base.cc:1034] running on GCE node
I20260927 22:48:05.425187 9483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:05.426074 9483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:05.427278 9483 hybrid_clock.cc:648] HybridClock initialized: now 1790549285427229 us; error 42 us; skew 500 ppm
I20260927 22:48:05.427618 9483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549285, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:05.430649 9483 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:48:05.432148 9483 webserver.cc:460] Webserver started at http://127.0.97.3:43991/ using document root <none> and password file <none>
I20260927 22:48:05.432829 9483 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:05.432948 9483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:05.438169 9483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:48:05.440809 9498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:05.442080 9483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:48:05.442225 9483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "82fc2b015c6840408727ae21c81d6922"
format_stamp: "Formatted at 2026-09-27 22:48:03 on dist-test-slave-kr25"
server_key: "dee5e3b1282f79dfbeaaaaf2a945cfa0"
server_key_iv: "c41e3bed613c7c928a1f355c2b9d2d01"
server_key_version: "encryptionkey@0"
I20260927 22:48:05.443114 9483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:05.466698 9483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:05.470602 9483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:05.470856 9483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:05.471835 9483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:05.473093 9483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:05.473189 9483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:05.473294 9483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:05.473361 9483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:05.502980 9483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46439
I20260927 22:48:05.503034 9610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46439 every 8 connection(s)
I20260927 22:48:05.504381 9483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:48:05.510201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 9483
W20260927 22:48:05.511071 9198 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.0.97.62:46439: Invalid argument: socket address is ambiguous
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:48:05.520313 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:05.506753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57155 (local address 127.0.97.62:46439)
0927 22:48:05.506943 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.506947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.508019 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 22:48:05.509830 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.509834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.509897 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.509980 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.511889 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.512877 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.514898 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.515351 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.515477 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.520055 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57155: BlockingRecv error: recv got EOF from 127.0.97.3:57155 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:48:05.520961 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.507139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:57155)
0927 22:48:05.507859 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:05.507910 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:05.509577 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:05.510230 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:05.510241 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:05.510878 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:05.511681 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.511708 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.513067 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.513072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.514774 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.514787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.515564 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.515571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.515781 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.516371 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:05.516392 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:05.519868 (+ 3476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":296,"threads_started":1}
W20260927 22:48:05.521637 9611 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46439 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46439: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260927 22:48:05.525862 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.511311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.0.97.62:46439)
0927 22:48:05.511832 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.511837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.512125 (+ 288us) server_negotiation.cc:407] Connection header received
0927 22:48:05.512266 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.512269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.512329 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.512407 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.514250 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.515082 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.516273 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.516659 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.516760 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.519333 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:05.519341 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:05.519357 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:05.519389 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:05.522713 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:05.523477 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:05.523485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:05.523494 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:05.523696 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:05.524111 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:05.524116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:05.524119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:05.524620 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:05.524831 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:05.525267 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 22:48:05.525506 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":248,"threads_started":1}
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:05.529840 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.522159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:44571)
0927 22:48:05.522315 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:05.522332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:05.522417 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:05.522707 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:05.522710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:05.523083 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:05.523252 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.523263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.524477 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.524480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.525718 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.525730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.526450 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.526455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.526614 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.527080 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:05.527096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:05.529663 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 22:48:05.529978 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:05.522274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44571 (local address 127.0.97.62:46439)
0927 22:48:05.522436 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.522441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.522458 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:48:05.522513 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.522519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.522586 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.522680 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.523379 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.524345 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.525841 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.526318 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.526409 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.529808 (+ 3399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44571: BlockingRecv error: recv got EOF from 127.0.97.3:44571 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:05.536995 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.530378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:57343)
0927 22:48:05.530529 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:05.530546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:05.530625 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:05.530831 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:05.530834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:05.531133 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:05.531261 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.531268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.532213 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.532216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.533167 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.533177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.533781 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.533785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.533927 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.534402 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:05.534418 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:05.536807 (+ 2389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 22:48:05.537158 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:05.530445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57343 (local address 127.0.97.62:46439)
0927 22:48:05.530634 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.530638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.530652 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:48:05.530699 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.530702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.530746 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.530808 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.531373 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.532103 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.533281 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.533665 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.533766 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.536965 (+ 3199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57343: BlockingRecv error: recv got EOF from 127.0.97.3:57343 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260927 22:48:05.537256 9611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:05 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:05.693086 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:05.685332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:33483)
0927 22:48:05.685759 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:05.685776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:05.685886 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:05.686137 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:05.686139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:05.686576 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:05.686752 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.686761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.687745 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.687748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.688666 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:05.688677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.689323 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.689327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:05.689441 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.689979 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:05.689995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:05.692856 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
W20260927 22:48:05.693207 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:05.685395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33483 (local address 127.0.97.62:46439)
0927 22:48:05.685599 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:05.685605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:05.685847 (+ 242us) server_negotiation.cc:407] Connection header received
0927 22:48:05.685972 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:05.685976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:05.686034 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:05.686109 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:05.686894 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.687643 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.688781 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:05.689189 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:05.689279 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:05.693041 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33483: BlockingRecv error: recv got EOF from 127.0.97.1:33483 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:48:06 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:06 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:06.114606 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:06.106424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:51219)
0927 22:48:06.106900 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:06.106916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:06.107060 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:06.107355 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:06.107360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:06.107854 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:06.108051 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:06.108060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.109017 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.109021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:06.109935 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:06.109947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.110639 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.110645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:06.110791 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:06.111361 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:06.111378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:06.114315 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
W20260927 22:48:06.114753 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:06.106480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51219 (local address 127.0.97.62:46439)
0927 22:48:06.106690 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:06.106697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:06.106990 (+ 293us) server_negotiation.cc:407] Connection header received
0927 22:48:06.107166 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:06.107171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:06.107231 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:06.107307 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:06.108182 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.108869 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.110080 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.110474 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.110582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:06.114518 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51219: BlockingRecv error: recv got EOF from 127.0.97.2:51219 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:48:06.177734 9195 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.62:46439
W20260927 22:48:06.506471 9607 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.3:46439
Sep 27 22:48:06 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:06 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:06.546085 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:06.537856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:59449)
0927 22:48:06.538312 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:06.538326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:06.538468 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:06.538764 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:06.538769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:06.539207 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:06.539397 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:06.539406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.540448 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.540452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:06.541577 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:06.541589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.542354 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.542360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:06.542510 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:06.543114 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:06.543132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:06.545821 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":246,"threads_started":1}
W20260927 22:48:06.546315 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:06.537904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59449 (local address 127.0.97.62:46439)
0927 22:48:06.538087 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:06.538093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:06.538425 (+ 332us) server_negotiation.cc:407] Connection header received
0927 22:48:06.538572 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:06.538577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:06.538644 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:06.538721 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:06.539566 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.540312 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.541732 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.542183 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.542291 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:06.546062 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59449: BlockingRecv error: recv got EOF from 127.0.97.3:59449 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:48:06 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:06 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:06.709105 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:06.693925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:33265)
0927 22:48:06.694362 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:06.694379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:06.694491 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:06.694753 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:06.694756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:06.695124 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:06.695265 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:06.695272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.696266 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.696270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:06.697147 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:06.697158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.697815 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.697819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:06.697925 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:06.706292 (+ 8367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:06.706314 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:06.708854 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":233,"threads_started":1}
W20260927 22:48:06.709286 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:06.694094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33265 (local address 127.0.97.62:46439)
0927 22:48:06.694314 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:06.694321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:06.694459 (+ 138us) server_negotiation.cc:407] Connection header received
0927 22:48:06.694574 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:06.694577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:06.694646 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:06.694721 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:06.695387 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.696171 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.697263 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:06.697669 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:06.697768 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:06.709056 (+ 11288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33265: BlockingRecv error: recv got EOF from 127.0.97.1:33265 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 22:48:07 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:07 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:07.122890 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:07.115506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:46505)
0927 22:48:07.115962 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:07.115978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:07.116089 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:07.116345 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:07.116348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:07.116747 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:07.116888 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:07.116896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.117809 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.117813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:07.118718 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:07.118729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.119487 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.119491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:07.119613 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:07.120186 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:07.120203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:07.122662 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
W20260927 22:48:07.123062 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:07.115542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46505 (local address 127.0.97.62:46439)
0927 22:48:07.115692 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:07.115697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:07.116040 (+ 343us) server_negotiation.cc:407] Connection header received
0927 22:48:07.116175 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:07.116178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:07.116238 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:07.116314 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:07.116993 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.117702 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.118893 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.119338 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.119469 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:07.122853 (+ 3384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46505: BlockingRecv error: recv got EOF from 127.0.97.2:46505 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 22:48:07 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:07 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:07.555392 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:07.546971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:45771)
0927 22:48:07.547509 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:07.547526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:07.547661 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:07.548020 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:07.548025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:07.548464 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:07.548622 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:07.548635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.549696 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.549703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:07.550669 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:07.550681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.551401 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.551407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:07.551595 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:07.552227 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:07.552246 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:07.555116 (+ 2870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":269,"threads_started":1}
W20260927 22:48:07.555596 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:07.547094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45771 (local address 127.0.97.62:46439)
0927 22:48:07.547285 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:07.547292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:07.547605 (+ 313us) server_negotiation.cc:407] Connection header received
0927 22:48:07.547774 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:07.547780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:07.547879 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:07.547971 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:07.548773 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.549551 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.550820 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.551251 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.551370 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:07.555314 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45771: BlockingRecv error: recv got EOF from 127.0.97.3:45771 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:48:07 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:07 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:07.717434 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:07.709987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:52145)
0927 22:48:07.710438 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:07.710453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:07.710584 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:07.710878 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:07.710883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:07.711226 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:07.711384 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:07.711393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.712568 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.712572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:07.713425 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:07.713436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.714118 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.714123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:07.714239 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:07.714734 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:07.714750 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:07.717190 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":253,"threads_started":1}
W20260927 22:48:07.717629 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:07.710080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52145 (local address 127.0.97.62:46439)
0927 22:48:07.710282 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:07.710289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:07.710528 (+ 239us) server_negotiation.cc:407] Connection header received
0927 22:48:07.710691 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:07.710696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:07.710756 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:07.710835 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:07.711552 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.712433 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.713568 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:07.713999 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:07.714087 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:07.717406 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52145: BlockingRecv error: recv got EOF from 127.0.97.1:52145 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:48:08 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:08 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:08.131358 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:08.123821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:44447)
0927 22:48:08.124289 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:08.124304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:08.124426 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:08.124741 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:08.124746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:08.125087 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:08.125251 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:08.125260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.126239 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.126245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:08.127191 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:08.127204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.127901 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.127907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:08.128067 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:08.128619 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:08.128634 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:08.131047 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":253,"threads_started":1}
W20260927 22:48:08.131486 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:08.123959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44447 (local address 127.0.97.62:46439)
0927 22:48:08.124158 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:08.124165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:08.124376 (+ 211us) server_negotiation.cc:407] Connection header received
0927 22:48:08.124553 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:08.124557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:08.124614 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:08.124695 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:08.125390 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.126119 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.127323 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.127771 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.127911 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:08.131245 (+ 3334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44447: BlockingRecv error: recv got EOF from 127.0.97.2:44447 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 22:48:08 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:08 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:08.563746 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:08.556283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:42401)
0927 22:48:08.556737 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:08.556753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:08.556869 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:08.557138 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:08.557141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:08.557454 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:08.557610 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:08.557618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.558542 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.558545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:08.559419 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:08.559464 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.560119 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.560124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:08.560261 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:08.560867 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:08.560885 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:08.563456 (+ 2571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
W20260927 22:48:08.563954 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:08.556376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42401 (local address 127.0.97.62:46439)
0927 22:48:08.556584 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:08.556591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:08.556824 (+ 233us) server_negotiation.cc:407] Connection header received
0927 22:48:08.556967 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:08.556971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:08.557029 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:08.557108 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:08.557716 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.558412 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.559575 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.559975 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.560074 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:08.563685 (+ 3611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42401: BlockingRecv error: recv got EOF from 127.0.97.3:42401 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:48:08 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:08 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:08.726317 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:08.718325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:46073)
0927 22:48:08.719059 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:08.719076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:08.719219 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:08.719536 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:08.719540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:08.719881 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:08.720037 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:08.720045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.720962 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.720966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:08.721906 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:08.721917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.722521 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.722525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:08.722654 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:08.723282 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:08.723306 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:08.726072 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":498,"threads_started":1}
W20260927 22:48:08.726485 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:08.718426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46073 (local address 127.0.97.62:46439)
0927 22:48:08.718661 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:08.718666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:08.719169 (+ 503us) server_negotiation.cc:407] Connection header received
0927 22:48:08.719314 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:08.719317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:08.719376 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:08.719484 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:08.720148 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.720842 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.722009 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:08.722397 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:08.722479 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:08.726277 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46073: BlockingRecv error: recv got EOF from 127.0.97.1:46073 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 22:48:09 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:09 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:09.141247 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:09.132250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:38177)
0927 22:48:09.132742 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:09.132759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:09.132983 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:09.133298 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:09.133304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:09.133718 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:09.133908 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:09.133918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.135003 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.135009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:09.136182 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:09.136195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.136917 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.136923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:09.137104 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:09.137713 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:09.137732 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:09.140891 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":273,"threads_started":1}
W20260927 22:48:09.141508 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:09.132345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38177 (local address 127.0.97.62:46439)
0927 22:48:09.132522 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:09.132530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:09.132905 (+ 375us) server_negotiation.cc:407] Connection header received
0927 22:48:09.133100 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:09.133106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:09.133169 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:09.133259 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:09.134090 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.134877 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.136331 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.136767 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.136882 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:09.141267 (+ 4385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38177: BlockingRecv error: recv got EOF from 127.0.97.2:38177 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 22:48:09 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:09 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:09.572139 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:09.564665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:59091)
0927 22:48:09.565154 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:09.565170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:09.565304 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:09.565600 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:09.565604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:09.565966 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:09.566161 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:09.566170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.567156 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.567161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:09.568147 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:09.568159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.568805 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.568811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:09.568966 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:09.569467 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:09.569483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:09.571922 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
W20260927 22:48:09.572296 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:09.564745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59091 (local address 127.0.97.62:46439)
0927 22:48:09.564937 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:09.564944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:09.565251 (+ 307us) server_negotiation.cc:407] Connection header received
0927 22:48:09.565413 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:09.565418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:09.565477 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:09.565572 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:09.566287 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.567032 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.568281 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.568679 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.568781 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:09.572095 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59091: BlockingRecv error: recv got EOF from 127.0.97.3:59091 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 22:48:09 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:09 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:09.734969 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:09.727209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:57165)
0927 22:48:09.727702 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:09.727718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:09.727829 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:09.728080 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:09.728083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:09.728455 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:09.728607 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:09.728615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.729540 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.729544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:09.730706 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:09.730718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.731453 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.731460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:09.731591 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:09.732154 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:09.732171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:09.734723 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":249,"threads_started":1}
W20260927 22:48:09.735092 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:09.727273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57165 (local address 127.0.97.62:46439)
0927 22:48:09.727516 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:09.727523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:09.727787 (+ 264us) server_negotiation.cc:407] Connection header received
0927 22:48:09.727919 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:09.727922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:09.727979 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:09.728056 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:09.728715 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.729419 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.730801 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:09.731304 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:09.731393 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:09.734895 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57165: BlockingRecv error: recv got EOF from 127.0.97.1:57165 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:48:10 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:10 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:10.150079 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:10.142274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:51915)
0927 22:48:10.142712 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:10.142728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:10.142830 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:10.143115 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:10.143118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:10.143510 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:10.143677 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:10.143685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.144605 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.144609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:10.145494 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:10.145504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.146122 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.146127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:10.146245 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:10.146818 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:10.146836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:10.149758 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
W20260927 22:48:10.150178 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:10.142433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51915 (local address 127.0.97.62:46439)
0927 22:48:10.142633 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:10.142639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:10.142819 (+ 180us) server_negotiation.cc:407] Connection header received
0927 22:48:10.142944 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:10.142948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:10.143010 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:10.143086 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:10.143779 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.144507 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.145604 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.146008 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.146091 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:10.149964 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51915: BlockingRecv error: recv got EOF from 127.0.97.2:51915 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 22:48:10 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:10 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:10.580788 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:10.573056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:57459)
0927 22:48:10.573641 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:10.573656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:10.573759 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:10.574079 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:10.574083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:10.574465 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:10.574622 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:10.574631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.575638 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.575642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:10.576535 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:10.576545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.577198 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.577204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:10.577350 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:10.577921 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:10.577943 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:10.580539 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":314,"threads_started":1}
W20260927 22:48:10.580957 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:10.573078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57459 (local address 127.0.97.62:46439)
0927 22:48:10.573249 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:10.573257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:10.573731 (+ 474us) server_negotiation.cc:407] Connection header received
0927 22:48:10.573883 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:10.573887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:10.573952 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:10.574027 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:10.574756 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.575515 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.576659 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.577070 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.577171 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:10.580698 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57459: BlockingRecv error: recv got EOF from 127.0.97.3:57459 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 22:48:10 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:10 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:10.743542 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:10.735901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:53107)
0927 22:48:10.736402 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:10.736438 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:10.736571 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:10.736872 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:10.736877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:10.737331 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:10.737481 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:10.737490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.738469 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.738475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:10.739349 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:10.739360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.740049 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.740055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:10.740214 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:10.740718 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:10.740733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:10.743289 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":274,"threads_started":1}
W20260927 22:48:10.743712 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:10.735947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53107 (local address 127.0.97.62:46439)
0927 22:48:10.736179 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:10.736186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:10.736501 (+ 315us) server_negotiation.cc:407] Connection header received
0927 22:48:10.736685 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:10.736690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:10.736749 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:10.736825 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:10.737609 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.738334 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.739499 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:10.739917 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:10.740018 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:10.743484 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53107: BlockingRecv error: recv got EOF from 127.0.97.1:53107 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:48:11 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:11 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:11.159124 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:11.151190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:48471)
0927 22:48:11.151662 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:11.151677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:11.151801 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:11.152119 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:11.152123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:11.152510 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:11.152672 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:11.152682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.153691 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.153695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:11.154602 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:11.154614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.155324 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.155330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:11.155531 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:11.156102 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:11.156120 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:11.158869 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":275,"threads_started":1}
W20260927 22:48:11.159375 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:11.151241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48471 (local address 127.0.97.62:46439)
0927 22:48:11.151412 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:11.151420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:11.151761 (+ 341us) server_negotiation.cc:407] Connection header received
0927 22:48:11.151910 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:11.151915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:11.151977 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:11.152059 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:11.152828 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.153537 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.154745 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.155166 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.155272 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:11.159130 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48471: BlockingRecv error: recv got EOF from 127.0.97.2:48471 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 22:48:11 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:11 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:11.590667 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:11.581769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:34695)
0927 22:48:11.582247 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:11.582263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:11.582383 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:11.582675 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:11.582679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:11.583073 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:11.583224 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:11.583233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.584471 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.584477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:11.585933 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:11.585951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.586836 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.586844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:11.587013 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:11.587731 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:11.587751 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:11.590437 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
W20260927 22:48:11.590816 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:11.581854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34695 (local address 127.0.97.62:46439)
0927 22:48:11.582030 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:11.582037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:11.582339 (+ 302us) server_negotiation.cc:407] Connection header received
0927 22:48:11.582496 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:11.582500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:11.582557 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:11.582633 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:11.583354 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.584311 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.586072 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.586653 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.586764 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:11.590621 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34695: BlockingRecv error: recv got EOF from 127.0.97.3:34695 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 22:48:11 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:11 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:11.756913 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:11.744472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:55257)
0927 22:48:11.745019 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:11.745036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:11.745158 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:11.745417 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:11.745420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:11.745766 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:11.745919 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:11.745926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.746976 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.746980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:11.747977 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:11.747989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.748675 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.748681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:11.748883 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:11.753941 (+ 5058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:11.753960 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:11.756677 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":234,"threads_started":1}
W20260927 22:48:11.757069 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:11.744575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55257 (local address 127.0.97.62:46439)
0927 22:48:11.744732 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:11.744737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:11.745139 (+ 402us) server_negotiation.cc:407] Connection header received
0927 22:48:11.745246 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:11.745249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:11.745310 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:11.745392 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:11.746040 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.746831 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.748100 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:11.748524 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:11.748628 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:11.756869 (+ 8241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55257: BlockingRecv error: recv got EOF from 127.0.97.1:55257 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 22:48:12 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:12 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:12.169354 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:12.160386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:43317)
0927 22:48:12.160981 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:12.160998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:12.161149 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:12.161532 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:12.161537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:12.162041 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:12.162261 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:12.162275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.163323 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.163328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:12.164399 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:12.164411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.165064 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.165070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:12.165220 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:12.165825 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:12.165842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:12.169013 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":251,"threads_started":1}
W20260927 22:48:12.169447 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:12.160457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43317 (local address 127.0.97.62:46439)
0927 22:48:12.160682 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:12.160688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:12.161068 (+ 380us) server_negotiation.cc:407] Connection header received
0927 22:48:12.161275 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:12.161281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:12.161363 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:12.161475 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:12.162397 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.163190 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.164527 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.164924 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.165045 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:12.169246 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43317: BlockingRecv error: recv got EOF from 127.0.97.2:43317 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 22:48:12 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:12 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:12.599303 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:12.591648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:47951)
0927 22:48:12.592152 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:12.592167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:12.592304 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:12.592602 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:12.592607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:12.593000 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:12.593182 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:12.593191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.594201 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.594207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:12.595108 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:12.595119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.595863 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.595868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:12.596018 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:12.596521 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:12.596538 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:12.599041 (+ 2503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":210,"threads_started":1}
W20260927 22:48:12.599480 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:12.591693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47951 (local address 127.0.97.62:46439)
0927 22:48:12.591843 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:12.591849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:12.592248 (+ 399us) server_negotiation.cc:407] Connection header received
0927 22:48:12.592417 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:12.592422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:12.592486 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:12.592563 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:12.593314 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.594071 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.595242 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.595722 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.595833 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:12.599264 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47951: BlockingRecv error: recv got EOF from 127.0.97.3:47951 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 22:48:12 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:12 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:12.765463 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:12.757768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:58911)
0927 22:48:12.758224 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:12.758240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:12.758348 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:12.758665 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:12.758668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:12.759069 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:12.759216 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:12.759223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.760234 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.760238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:12.761116 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:12.761126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.761798 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.761802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:12.761912 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:12.762425 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:12.762448 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:12.765167 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":275,"threads_started":1}
W20260927 22:48:12.765621 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:12.757949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58911 (local address 127.0.97.62:46439)
0927 22:48:12.758143 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:12.758150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:12.758311 (+ 161us) server_negotiation.cc:407] Connection header received
0927 22:48:12.758479 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:12.758483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:12.758543 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:12.758630 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:12.759320 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.760125 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.761234 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:12.761669 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:12.761756 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:12.765372 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58911: BlockingRecv error: recv got EOF from 127.0.97.1:58911 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:48:13 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:13 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:13.178553 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:13.170311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:34433)
0927 22:48:13.170789 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:13.170805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:13.170912 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:13.171177 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:13.171180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:13.171596 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:13.171739 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:13.171746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.172820 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.172824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:13.173856 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:13.173868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.174533 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.174537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:13.174651 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:13.175295 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:13.175317 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:13.178262 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
W20260927 22:48:13.178596 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:13.170424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34433 (local address 127.0.97.62:46439)
0927 22:48:13.170585 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:13.170590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:13.170866 (+ 276us) server_negotiation.cc:407] Connection header received
0927 22:48:13.171001 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:13.171005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:13.171063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:13.171144 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:13.171853 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.172708 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.173957 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.174377 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.174473 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:13.178425 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34433: BlockingRecv error: recv got EOF from 127.0.97.2:34433 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 22:48:13 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:13 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:13.607658 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:13.600197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:40835)
0927 22:48:13.600695 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:13.600713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:13.600833 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:13.601181 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:13.601185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:13.601547 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:13.601703 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:13.601711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.602647 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.602650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:13.603555 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:13.603566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.604213 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.604218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:13.604333 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:13.604838 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:13.604855 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:13.607388 (+ 2533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":279,"threads_started":1}
W20260927 22:48:13.607844 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:13.600258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40835 (local address 127.0.97.62:46439)
0927 22:48:13.600423 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:13.600427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:13.600777 (+ 350us) server_negotiation.cc:407] Connection header received
0927 22:48:13.600946 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:13.600950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:13.601015 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:13.601092 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:13.601811 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.602533 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.603684 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.604081 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.604168 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:13.607629 (+ 3461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40835: BlockingRecv error: recv got EOF from 127.0.97.3:40835 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 22:48:13 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:13 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:13.773502 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:13.766307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:58453)
0927 22:48:13.766766 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:13.766782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:13.766889 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:13.767180 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:13.767183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:13.767564 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:13.767724 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:13.767732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.768676 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.768679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:13.769597 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:13.769608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.770206 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.770209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:13.770318 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:13.770811 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:13.770826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:13.773276 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
W20260927 22:48:13.773720 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:13.766351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58453 (local address 127.0.97.62:46439)
0927 22:48:13.766550 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:13.766558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:13.766853 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:48:13.766999 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:13.767003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:13.767072 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:13.767146 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:13.767831 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.768580 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.769702 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:13.770101 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:13.770196 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:13.773472 (+ 3276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58453: BlockingRecv error: recv got EOF from 127.0.97.1:58453 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:48:14 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:14 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:14.187733 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:14.179492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:35155)
0927 22:48:14.180039 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:14.180055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:14.180174 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:14.180494 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:14.180498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:14.180870 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:14.181053 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:14.181063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.182395 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.182398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:14.183400 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:14.183413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.184186 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.184192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:14.184359 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:14.184921 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:14.184938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:14.187487 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":323,"threads_started":1}
W20260927 22:48:14.187858 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:14.179598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35155 (local address 127.0.97.62:46439)
0927 22:48:14.179767 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:14.179774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:14.180130 (+ 356us) server_negotiation.cc:407] Connection header received
0927 22:48:14.180271 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:14.180276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:14.180349 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:14.180468 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:14.181177 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.182291 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.183542 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.184026 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.184162 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:14.187655 (+ 3493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35155: BlockingRecv error: recv got EOF from 127.0.97.2:35155 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 22:48:14 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:14 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:14.616235 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:14.608462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:39283)
0927 22:48:14.608959 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:14.608975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:14.609098 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:14.609411 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:14.609415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:14.609767 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:14.609958 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:14.609968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.610900 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.610904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:14.611871 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:14.611890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.612557 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.612563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:14.612718 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:14.613232 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:14.613253 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:14.615995 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
W20260927 22:48:14.616417 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:14.608531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39283 (local address 127.0.97.62:46439)
0927 22:48:14.608715 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:14.608722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:14.609048 (+ 326us) server_negotiation.cc:407] Connection header received
0927 22:48:14.609211 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:14.609215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:14.609289 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:14.609364 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:14.610087 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.610775 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.612004 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.612426 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.612528 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:14.616201 (+ 3673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39283: BlockingRecv error: recv got EOF from 127.0.97.3:39283 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:48:14 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:14 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:14.781958 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:14.774391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:35593)
0927 22:48:14.774842 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:14.774858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:14.774973 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:14.775248 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:14.775251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:14.775647 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:14.775793 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:14.775801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.776728 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.776732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:14.777623 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:14.777633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.778263 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.778267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:14.778392 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:14.778912 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:14.778928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:14.781703 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":262,"threads_started":1}
W20260927 22:48:14.782084 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:14.774524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35593 (local address 127.0.97.62:46439)
0927 22:48:14.774731 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:14.774737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:14.774930 (+ 193us) server_negotiation.cc:407] Connection header received
0927 22:48:14.775061 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:14.775064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:14.775143 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:14.775219 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:14.775904 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.776624 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.777721 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:14.778143 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:14.778229 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:14.781893 (+ 3664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35593: BlockingRecv error: recv got EOF from 127.0.97.1:35593 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 22:48:15 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:15 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:15.196493 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:15.188562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:35559)
0927 22:48:15.189039 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:15.189056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:15.189178 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:15.189435 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:15.189438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:15.189795 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:15.189940 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:15.189948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.190922 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.190928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:15.192241 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:15.192259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.192965 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.192969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:15.193081 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:15.193637 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:15.193653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:15.196266 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":261,"threads_started":1}
W20260927 22:48:15.196789 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:15.188631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35559 (local address 127.0.97.62:46439)
0927 22:48:15.188768 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:15.188772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:15.189140 (+ 368us) server_negotiation.cc:407] Connection header received
0927 22:48:15.189264 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:15.189268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:15.189325 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:15.189403 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:15.190092 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.190788 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.192376 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.192832 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.192949 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:15.196440 (+ 3491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35559: BlockingRecv error: recv got EOF from 127.0.97.2:35559 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 22:48:15 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:15 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:15.624207 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:15.617117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:32933)
0927 22:48:15.617573 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:15.617588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:15.617691 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:15.617935 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:15.617938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:15.618280 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:15.618426 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:15.618433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.619360 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.619363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:15.620283 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:15.620295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.620888 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.620892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:15.621003 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:15.621553 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:15.621569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:15.623988 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
W20260927 22:48:15.624358 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:15.617207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32933 (local address 127.0.97.62:46439)
0927 22:48:15.617366 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:15.617370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:15.617651 (+ 281us) server_negotiation.cc:407] Connection header received
0927 22:48:15.617775 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:15.617778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:15.617833 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:15.617909 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:15.618544 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.619263 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.620391 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.620781 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.620865 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:15.624152 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:32933: BlockingRecv error: recv got EOF from 127.0.97.3:32933 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:48:15 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:15 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:15.790401 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:15.782773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:40973)
0927 22:48:15.783261 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:15.783278 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:15.783386 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:15.783669 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:15.783672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:15.784036 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:15.784197 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:15.784205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.785078 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.785081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:15.785952 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:15.785971 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.786659 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.786665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:15.786808 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:15.787428 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:15.787478 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:15.790158 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
W20260927 22:48:15.790521 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:15.782837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40973 (local address 127.0.97.62:46439)
0927 22:48:15.783000 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:15.783004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:15.783338 (+ 334us) server_negotiation.cc:407] Connection header received
0927 22:48:15.783507 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:15.783510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:15.783579 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:15.783656 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:15.784298 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.784979 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.786058 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:15.786491 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:15.786591 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:15.790326 (+ 3735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40973: BlockingRecv error: recv got EOF from 127.0.97.1:40973 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:48:16 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:16 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:16.204913 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:16.197350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:50981)
0927 22:48:16.197846 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:16.197864 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:16.197974 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:16.198250 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:16.198253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:16.198626 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:16.198797 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:16.198806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.199881 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.199885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:16.200767 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:16.200778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.201447 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.201451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:16.201555 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:16.202117 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:16.202134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:16.204685 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
W20260927 22:48:16.205099 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:16.197419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50981 (local address 127.0.97.62:46439)
0927 22:48:16.197574 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:16.197579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:16.197931 (+ 352us) server_negotiation.cc:407] Connection header received
0927 22:48:16.198081 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:16.198085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:16.198144 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:16.198220 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:16.198924 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.199771 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.200892 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.201309 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.201399 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:16.204868 (+ 3469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50981: BlockingRecv error: recv got EOF from 127.0.97.2:50981 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:48:16 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:16 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:16.632503 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:16.624967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:43247)
0927 22:48:16.625449 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:16.625465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:16.625584 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:16.625863 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:16.625868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:16.626285 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:16.626446 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:16.626455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.627415 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.627420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:16.628330 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:16.628341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.629009 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.629015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:16.629165 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:16.629725 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:16.629742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:16.632280 (+ 2538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":248,"threads_started":1}
W20260927 22:48:16.632635 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:16.625028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43247 (local address 127.0.97.62:46439)
0927 22:48:16.625203 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:16.625209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:16.625542 (+ 333us) server_negotiation.cc:407] Connection header received
0927 22:48:16.625693 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:16.625697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:16.625754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:16.625829 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:16.626583 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.627299 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.628458 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.628864 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.628970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:16.632448 (+ 3478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43247: BlockingRecv error: recv got EOF from 127.0.97.3:43247 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:48:16 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:16 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:16.803157 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:16.791312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:49059)
0927 22:48:16.791861 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:16.791877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:16.792015 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:16.792344 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:16.792348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:16.792737 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:16.792894 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:16.792903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.793904 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.793908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:16.794824 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:16.794835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.795608 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.795614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:16.795779 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:16.800038 (+ 4259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:16.800056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:16.802916 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
W20260927 22:48:16.803336 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:16.791474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49059 (local address 127.0.97.62:46439)
0927 22:48:16.791679 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:16.791687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:16.791960 (+ 273us) server_negotiation.cc:407] Connection header received
0927 22:48:16.792126 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:16.792132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:16.792212 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:16.792291 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:16.793043 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.793772 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.794969 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:16.795473 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:16.795584 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:16.803110 (+ 7526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49059: BlockingRecv error: recv got EOF from 127.0.97.1:49059 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 22:48:17 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:17 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:17.213394 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:17.205752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:59865)
0927 22:48:17.206247 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:17.206263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:17.206380 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:17.206701 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:17.206705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:17.207171 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:17.207330 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:17.207342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.208466 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.208470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:17.209366 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:17.209377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.210030 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.210034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:17.210174 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:17.210697 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:17.210713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:17.213173 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":267,"threads_started":1}
W20260927 22:48:17.213572 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:17.205857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59865 (local address 127.0.97.62:46439)
0927 22:48:17.206011 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:17.206016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:17.206351 (+ 335us) server_negotiation.cc:407] Connection header received
0927 22:48:17.206485 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:17.206491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:17.206554 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:17.206656 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:17.207521 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.208333 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.209488 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.209905 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.209998 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:17.213368 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59865: BlockingRecv error: recv got EOF from 127.0.97.2:59865 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 22:48:17 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:17 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:17.641371 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:17.633319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:55265)
0927 22:48:17.633810 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:17.633825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:17.633948 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:17.634272 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:17.634276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:17.634699 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:17.634844 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:17.634852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.635941 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.635947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:17.636836 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:17.636846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.637548 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.637554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:17.637722 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:17.638321 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:17.638339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:17.641054 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
W20260927 22:48:17.641528 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:17.633354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55265 (local address 127.0.97.62:46439)
0927 22:48:17.633519 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:17.633526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:17.633901 (+ 375us) server_negotiation.cc:407] Connection header received
0927 22:48:17.634062 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:17.634067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:17.634154 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:17.634230 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:17.634975 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.635812 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.636965 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.637419 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.637532 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:17.641283 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55265: BlockingRecv error: recv got EOF from 127.0.97.3:55265 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 22:48:17 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:17 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:17.812378 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:17.804248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:53641)
0927 22:48:17.804759 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:17.804774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:17.804884 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:17.805229 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:17.805234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:17.805625 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:17.805773 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:17.805781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.806851 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.806856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:17.808016 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:17.808027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.808797 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.808802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:17.808938 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:17.809481 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:17.809500 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:17.812161 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
W20260927 22:48:17.812561 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:17.804321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53641 (local address 127.0.97.62:46439)
0927 22:48:17.804463 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:17.804468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:17.804869 (+ 401us) server_negotiation.cc:407] Connection header received
0927 22:48:17.805017 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:17.805020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:17.805086 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:17.805178 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:17.805911 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.806715 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.808135 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:17.808618 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:17.808709 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:17.812318 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53641: BlockingRecv error: recv got EOF from 127.0.97.1:53641 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 22:48:18 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:18 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:18.222040 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:18.214245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:46173)
0927 22:48:18.214728 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:18.214743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:18.214870 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:18.215244 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:18.215248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:18.215654 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:18.215808 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:18.215816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.216780 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.216784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:18.217728 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:18.217740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.218406 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.218411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:18.218529 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:18.219088 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:18.219105 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:18.221789 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":254,"threads_started":1}
W20260927 22:48:18.222193 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:18.214451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46173 (local address 127.0.97.62:46439)
0927 22:48:18.214631 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:18.214637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:18.214823 (+ 186us) server_negotiation.cc:407] Connection header received
0927 22:48:18.215055 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:18.215060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:18.215132 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:18.215207 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:18.215922 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.216662 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.217856 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.218280 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.218367 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:18.221999 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46173: BlockingRecv error: recv got EOF from 127.0.97.2:46173 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:48:18 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:18 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:18.650671 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:18.642452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:54007)
0927 22:48:18.642964 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:18.642980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:18.643100 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:18.643379 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:18.643395 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:18.643791 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:18.643990 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:18.644001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.645062 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.645067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:18.646097 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:18.646110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.647064 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.647070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:18.647222 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:18.647864 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:18.647887 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:18.650416 (+ 2529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":260,"threads_started":1}
W20260927 22:48:18.650811 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:18.642464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54007 (local address 127.0.97.62:46439)
0927 22:48:18.642633 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:18.642639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:18.643057 (+ 418us) server_negotiation.cc:407] Connection header received
0927 22:48:18.643191 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:18.643194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:18.643254 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:18.643330 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:18.644144 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.644897 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.646236 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.646865 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.646992 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:18.650628 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54007: BlockingRecv error: recv got EOF from 127.0.97.3:54007 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:48:18 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:18 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:18.821408 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:18.813180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:57793)
0927 22:48:18.813702 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:18.813721 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:18.813867 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:18.814206 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:18.814210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:18.814643 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:18.814809 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:18.814818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.815868 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.815873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:18.816760 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:18.816771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.817412 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.817418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:18.817592 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:18.818175 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:18.818200 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:18.821155 (+ 2955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":298,"threads_started":1}
W20260927 22:48:18.821581 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:18.813312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57793 (local address 127.0.97.62:46439)
0927 22:48:18.813657 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:18.813664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:18.813790 (+ 126us) server_negotiation.cc:407] Connection header received
0927 22:48:18.814018 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:18.814023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:18.814082 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:18.814162 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:18.814940 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.815736 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.816883 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:18.817283 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:18.817387 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:18.821357 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57793: BlockingRecv error: recv got EOF from 127.0.97.1:57793 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Sep 27 22:48:19 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:19 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:19.231041 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:19.222910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:56919)
0927 22:48:19.223407 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:19.223424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:19.223563 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:19.223824 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:19.223827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:19.224161 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:19.224329 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:19.224337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.225329 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.225333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:19.226493 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:19.226506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.227306 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.227311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:19.227424 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:19.228018 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:19.228034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:19.230814 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":217,"threads_started":1}
W20260927 22:48:19.231199 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:19.223002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56919 (local address 127.0.97.62:46439)
0927 22:48:19.223181 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:19.223186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:19.223542 (+ 356us) server_negotiation.cc:407] Connection header received
0927 22:48:19.223648 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:19.223652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:19.223715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:19.223798 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:19.224469 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.225196 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.226611 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.227187 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.227276 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:19.231015 (+ 3739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56919: BlockingRecv error: recv got EOF from 127.0.97.2:56919 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:48:19 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:19 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:19.659456 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:19.651614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:49693)
0927 22:48:19.652048 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:19.652063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:19.652164 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:19.652443 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:19.652445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:19.652775 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:19.652936 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:19.652943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.653989 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.654019 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:19.654920 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:19.654931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.655655 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.655659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:19.655775 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:19.656391 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:19.656408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:19.659195 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":207,"threads_started":1}
W20260927 22:48:19.659730 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:19.651707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49693 (local address 127.0.97.62:46439)
0927 22:48:19.651886 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:19.651891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:19.652126 (+ 235us) server_negotiation.cc:407] Connection header received
0927 22:48:19.652249 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:19.652253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:19.652314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:19.652423 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:19.653082 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.653849 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.655048 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.655541 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.655646 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:19.659481 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49693: BlockingRecv error: recv got EOF from 127.0.97.3:49693 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:48:19 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:19 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:19.830047 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:19.822410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:51329)
0927 22:48:19.822891 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:19.822906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:19.823032 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:19.823342 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:19.823347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:19.823901 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:19.824086 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:19.824094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.825000 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.825004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:19.825874 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:19.825884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.826528 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.826531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:19.826633 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:19.827176 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:19.827193 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:19.829722 (+ 2529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
W20260927 22:48:19.830132 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:19.822479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51329 (local address 127.0.97.62:46439)
0927 22:48:19.822649 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:19.822655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:19.822979 (+ 324us) server_negotiation.cc:407] Connection header received
0927 22:48:19.823133 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:19.823137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:19.823202 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:19.823303 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:19.824210 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.824899 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.825988 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:19.826425 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:19.826523 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:19.829913 (+ 3390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51329: BlockingRecv error: recv got EOF from 127.0.97.1:51329 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 22:48:20 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:48:20 dist-test-slave-kr25 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:48:20.240299 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:20.232068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:52019)
0927 22:48:20.232554 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:20.232569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:20.232686 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:20.232976 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:20.232980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:20.233404 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:20.233591 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:20.233600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.234674 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.234678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:20.235614 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:20.235625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.236322 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.236328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:20.236486 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:20.237209 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:20.237232 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:20.240060 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46439: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
W20260927 22:48:20.240520 9350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:20.232182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52019 (local address 127.0.97.62:46439)
0927 22:48:20.232389 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:20.232395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:20.232643 (+ 248us) server_negotiation.cc:407] Connection header received
0927 22:48:20.232789 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:20.232792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:20.232848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:20.232923 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:20.233719 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.234543 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.235736 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.236187 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.236288 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:20.240291 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52019: BlockingRecv error: recv got EOF from 127.0.97.2:52019 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:48:20.520694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9201
I20260927 22:48:20.532413 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:20.521835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.0.97.1:41815)
0927 22:48:20.522337 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:20.522345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:20.522373 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:48:20.522445 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:20.522452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:20.522694 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:20.522856 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:20.524384 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.525383 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.526607 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.527198 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.527296 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:20.529654 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:20.529662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:20.529676 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:20.529713 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:20.531954 (+ 2241us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:48:20.532196 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57150: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":249,"threads_started":1}
W20260927 22:48:20.532512 9661 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57150: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:48:20.532557 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9201: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:41815: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:48:20.532804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 9201
I20260927 22:48:20.546134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9352
I20260927 22:48:20.558173 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:20.547051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.0.97.2:40109)
0927 22:48:20.547682 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:20.547689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:20.547718 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:48:20.547771 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:20.547778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:20.547998 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:20.548143 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:20.549354 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.550459 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.551796 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.552385 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.552496 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:20.554950 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:20.554961 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:20.554976 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:20.555012 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:20.557588 (+ 2576us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:48:20.557870 (+ 282us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52190: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":294,"threads_started":1}
W20260927 22:48:20.558290 9662 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52190: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:48:20.558277 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9352: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40109: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:48:20.558434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 9352
I20260927 22:48:20.572453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9483
I20260927 22:48:20.583535 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:20.573410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.0.97.3:46439)
0927 22:48:20.573968 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:20.573975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:20.574001 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:48:20.574059 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:20.574065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:20.574278 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:20.574430 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:20.575644 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.576685 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.577778 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:20.578287 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:20.578379 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:20.580689 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:20.580697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:20.580710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:20.580744 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:20.583016 (+ 2272us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:48:20.583241 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51966: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
W20260927 22:48:20.583588 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9483: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46439: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:48:20.583630 9663 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51966: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:48:20.583851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 9483
I20260927 22:48:20.596308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9132
I20260927 22:48:20.597442 9194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:20.708688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 9132
2026-09-27T22:48:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20696 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T22:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:48:20Z Disabled control of system clock
I20260927 22:48:20.773993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34943
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:48:21.077129 9668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:21.077394 9668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:21.077469 9668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:21.084492 9668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:48:21.084604 9668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:21.084640 9668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:21.084692 9668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:48:21.084745 9668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:48:21.093652 9668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:21.094834 9668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:21.096498 9668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:21.107510 9674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:21.107470 9676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:21.107556 9673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:21.108001 9668 server_base.cc:1034] running on GCE node
I20260927 22:48:21.108723 9668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:21.109993 9668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:21.111214 9668 hybrid_clock.cc:648] HybridClock initialized: now 1790549301111162 us; error 37 us; skew 500 ppm
I20260927 22:48:21.111585 9668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:21.113745 9668 webserver.cc:460] Webserver started at http://127.0.97.62:39749/ using document root <none> and password file <none>
I20260927 22:48:21.114375 9668 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:21.114476 9668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:21.114789 9668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:21.117471 9668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "13f0e481af2a43c1b51fc5ca37c04a29"
format_stamp: "Formatted at 2026-09-27 22:48:21 on dist-test-slave-kr25"
server_key: "ba83a5ab338343aae63cb14236f1e2be"
server_key_iv: "49700eb687767ee62e2b24cd67bf9487"
server_key_version: "encryptionkey@0"
I20260927 22:48:21.118227 9668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "13f0e481af2a43c1b51fc5ca37c04a29"
format_stamp: "Formatted at 2026-09-27 22:48:21 on dist-test-slave-kr25"
server_key: "ba83a5ab338343aae63cb14236f1e2be"
server_key_iv: "49700eb687767ee62e2b24cd67bf9487"
server_key_version: "encryptionkey@0"
I20260927 22:48:21.123150 9668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:48:21.126629 9682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:21.127992 9668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:48:21.128163 9668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "13f0e481af2a43c1b51fc5ca37c04a29"
format_stamp: "Formatted at 2026-09-27 22:48:21 on dist-test-slave-kr25"
server_key: "ba83a5ab338343aae63cb14236f1e2be"
server_key_iv: "49700eb687767ee62e2b24cd67bf9487"
server_key_version: "encryptionkey@0"
I20260927 22:48:21.128367 9668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:21.149528 9668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:21.150429 9668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:21.150722 9668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:21.168500 9668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34943
I20260927 22:48:21.168471 9733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34943 every 8 connection(s)
I20260927 22:48:21.169881 9668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:48:21.171412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 9668
I20260927 22:48:21.171622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:48:21.172420 388 external_mini_cluster.cc:1428] Setting key 90a98f8119a96980cc169b681cdbc894
I20260927 22:48:21.174542 9734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:48:21.186451 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:21.174599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.0.97.62:34943)
0927 22:48:21.175830 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:21.175841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:21.175895 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:48:21.177624 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:21.177645 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:21.178116 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:21.178472 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:21.179040 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:21.180474 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:21.182292 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:21.182954 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:21.183087 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:21.184301 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:21.184309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:21.184322 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:21.184875 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:21.184948 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:21.184970 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:21.185058 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:21.185273 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 22:48:21.185583 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":417,"threads_started":1}
I20260927 22:48:21.187309 9734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29: Bootstrap starting.
I20260927 22:48:21.190384 9734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29: Neither blocks nor log segments found. Creating new log.
I20260927 22:48:21.191278 9734 log.cc:824] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29: Log is configured to *not* fsync() on all Append() calls
I20260927 22:48:21.193809 9734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29: No bootstrap required, opened a new log
I20260927 22:48:21.200683 9734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34943 } }
I20260927 22:48:21.201006 9734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:48:21.201102 9734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f0e481af2a43c1b51fc5ca37c04a29, State: Initialized, Role: FOLLOWER
I20260927 22:48:21.201679 9734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34943 } }
I20260927 22:48:21.201879 9734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:48:21.202006 9734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:48:21.202149 9734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:48:21.204167 9734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34943 } }
I20260927 22:48:21.204619 9734 leader_election.cc:302] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13f0e481af2a43c1b51fc5ca37c04a29; no voters:
I20260927 22:48:21.205199 9734 leader_election.cc:288] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:48:21.205371 9739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:48:21.206028 9739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [term 1 LEADER]: Becoming Leader. State: Replica: 13f0e481af2a43c1b51fc5ca37c04a29, State: Running, Role: LEADER
I20260927 22:48:21.206511 9739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34943 } }
I20260927 22:48:21.207193 9734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:48:21.208973 9740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34943 } } }
I20260927 22:48:21.209425 9740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:21.209329 9741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13f0e481af2a43c1b51fc5ca37c04a29. Latest consensus state: current_term: 1 leader_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f0e481af2a43c1b51fc5ca37c04a29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34943 } } }
I20260927 22:48:21.209859 9741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29 [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:21.210836 9744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:48:21.215169 9744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:48:21.222882 9744 catalog_manager.cc:1347] Generated new cluster ID: 1a187aa28ec5494b9cc45d281277d17a
I20260927 22:48:21.222991 9744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:48:21.231407 9744 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:48:21.232653 9744 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:48:21.241971 9744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13f0e481af2a43c1b51fc5ca37c04a29: Generated new TSK 0
I20260927 22:48:21.242992 9744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:48:21.259130 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34943
--builtin_ntp_servers=127.0.97.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 22:48:21.619503 9758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:21.619773 9758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:21.619855 9758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:21.626972 9758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:21.627066 9758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:21.627174 9758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:48:21.636336 9758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:21.637514 9758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:21.639178 9758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:21.651257 9766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:21.651324 9764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:21.651536 9763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:21.652191 9758 server_base.cc:1034] running on GCE node
I20260927 22:48:21.652727 9758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:21.653563 9758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:21.654770 9758 hybrid_clock.cc:648] HybridClock initialized: now 1790549301654708 us; error 53 us; skew 500 ppm
I20260927 22:48:21.655059 9758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:21.657138 9758 webserver.cc:460] Webserver started at http://127.0.97.1:37471/ using document root <none> and password file <none>
I20260927 22:48:21.657773 9758 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:21.657866 9758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:21.658154 9758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:21.660832 9758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "58f05b5b3238487c8bfd850d9ef8ab83"
format_stamp: "Formatted at 2026-09-27 22:48:21 on dist-test-slave-kr25"
server_key: "2309102a6c766ab05a0339506214134d"
server_key_iv: "86dafca7f9ecad66bc9fb28a4f3f6e62"
server_key_version: "encryptionkey@0"
I20260927 22:48:21.661617 9758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "58f05b5b3238487c8bfd850d9ef8ab83"
format_stamp: "Formatted at 2026-09-27 22:48:21 on dist-test-slave-kr25"
server_key: "2309102a6c766ab05a0339506214134d"
server_key_iv: "86dafca7f9ecad66bc9fb28a4f3f6e62"
server_key_version: "encryptionkey@0"
I20260927 22:48:21.666272 9758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:48:21.669386 9772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:21.670625 9758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:48:21.670763 9758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "58f05b5b3238487c8bfd850d9ef8ab83"
format_stamp: "Formatted at 2026-09-27 22:48:21 on dist-test-slave-kr25"
server_key: "2309102a6c766ab05a0339506214134d"
server_key_iv: "86dafca7f9ecad66bc9fb28a4f3f6e62"
server_key_version: "encryptionkey@0"
I20260927 22:48:21.670934 9758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:21.690562 9758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:21.691322 9758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:21.691627 9758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:21.692483 9758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:21.693967 9758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:21.694061 9758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:21.694159 9758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:21.694222 9758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:21.722921 9758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46645
I20260927 22:48:21.722985 9884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46645 every 8 connection(s)
I20260927 22:48:21.724134 9758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:48:21.732336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 9758
I20260927 22:48:21.732522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:48:21.733314 388 external_mini_cluster.cc:1428] Setting key 09233a00465c409a7029137a483e3967
I20260927 22:48:21.735991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34943
--builtin_ntp_servers=127.0.97.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:48:21.736080 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:21.726451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58995 (local address 127.0.97.62:34943)
0927 22:48:21.726889 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:21.726895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:21.728040 (+ 1145us) server_negotiation.cc:407] Connection header received
0927 22:48:21.729646 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:21.729649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:21.729710 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:21.729799 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:21.730705 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:21.731380 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:21.733516 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:21.733914 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:21.734020 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:21.735363 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:21.735368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:21.735371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:21.735387 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:21.735461 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:21.735478 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:21.735549 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:21.735753 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:48:21.735823 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":198,"threads_started":1}
I20260927 22:48:21.737277 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:21.727015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34943 (local address 127.0.97.1:58995)
0927 22:48:21.727894 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:21.727931 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:21.729432 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:21.729965 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:21.729972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:21.730052 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:21.730584 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:21.730600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:21.731521 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:21.731524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:21.733360 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:21.733375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:21.734145 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:21.734152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:21.734358 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:21.735049 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:48:21.735052 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:21.735100 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:48:21.735194 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:21.735559 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:21.735569 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:21.735576 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:48:21.735751 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 22:48:21.736116 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":246,"threads_started":1}
I20260927 22:48:21.738575 9885 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34943
I20260927 22:48:21.738826 9885 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:21.739565 9885 heartbeater.cc:511] Master 127.0.97.62:34943 requested a full tablet report, sending...
I20260927 22:48:21.741448 9699 ts_manager.cc:194] Registered new tserver with Master: 58f05b5b3238487c8bfd850d9ef8ab83 (127.0.97.1:46645)
I20260927 22:48:21.743217 9699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:58995
W20260927 22:48:22.038089 9889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:22.038364 9889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:22.038447 9889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:22.045622 9889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:22.045728 9889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:22.045828 9889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:48:22.054638 9889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:22.055809 9889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:22.057425 9889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:22.069003 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:22.069031 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:22.069206 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:22.069610 9889 server_base.cc:1034] running on GCE node
I20260927 22:48:22.070150 9889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:22.071004 9889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:22.072189 9889 hybrid_clock.cc:648] HybridClock initialized: now 1790549302072149 us; error 27 us; skew 500 ppm
I20260927 22:48:22.072465 9889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:22.078152 9889 webserver.cc:460] Webserver started at http://127.0.97.2:43583/ using document root <none> and password file <none>
I20260927 22:48:22.078828 9889 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:22.078923 9889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:22.079223 9889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:22.081985 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "c212303dfc304054872b2ef61e48d398"
format_stamp: "Formatted at 2026-09-27 22:48:22 on dist-test-slave-kr25"
server_key: "b9fc680f536120f335b456a83462c5a5"
server_key_iv: "94fb3a88e7dddcabf45ce710b7831425"
server_key_version: "encryptionkey@0"
I20260927 22:48:22.082706 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c212303dfc304054872b2ef61e48d398"
format_stamp: "Formatted at 2026-09-27 22:48:22 on dist-test-slave-kr25"
server_key: "b9fc680f536120f335b456a83462c5a5"
server_key_iv: "94fb3a88e7dddcabf45ce710b7831425"
server_key_version: "encryptionkey@0"
I20260927 22:48:22.087599 9889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:48:22.090778 9903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:22.092042 9889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:48:22.092181 9889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "c212303dfc304054872b2ef61e48d398"
format_stamp: "Formatted at 2026-09-27 22:48:22 on dist-test-slave-kr25"
server_key: "b9fc680f536120f335b456a83462c5a5"
server_key_iv: "94fb3a88e7dddcabf45ce710b7831425"
server_key_version: "encryptionkey@0"
I20260927 22:48:22.092362 9889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:22.123135 9889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:22.123988 9889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:22.124233 9889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:22.125160 9889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:22.126588 9889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:22.126679 9889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:22.126787 9889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:22.126854 9889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:22.155498 9889 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32901
I20260927 22:48:22.155565 10015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32901 every 8 connection(s)
I20260927 22:48:22.156726 9889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:48:22.166432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 9889
I20260927 22:48:22.166616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:48:22.167155 388 external_mini_cluster.cc:1428] Setting key 93d64225794b0ad91f9e7c821e48ef8f
I20260927 22:48:22.169093 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:22.159082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45873 (local address 127.0.97.62:34943)
0927 22:48:22.159259 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:22.159263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:22.160780 (+ 1517us) server_negotiation.cc:407] Connection header received
0927 22:48:22.162554 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:22.162558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:22.162623 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:22.162711 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:22.163845 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.164561 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.166130 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.166726 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.166858 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:22.168276 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:22.168282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:22.168287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:22.168304 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:22.168359 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:22.168375 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:22.168476 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:22.168712 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 22:48:22.168795 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:48:22.169919 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34943
--builtin_ntp_servers=127.0.97.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:48:22.170501 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:22.159716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34943 (local address 127.0.97.2:45873)
0927 22:48:22.160615 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:22.160658 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:22.162306 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:22.162921 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:22.162970 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:22.163061 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:22.163668 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:22.163686 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.164697 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.164701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:22.165965 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:22.165976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.166969 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.166976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:22.167178 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:22.167905 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:48:22.167909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:22.167960 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:48:22.168098 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:22.168480 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:22.168490 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:22.168498 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:48:22.168670 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:48:22.169282 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":291,"threads_started":1}
I20260927 22:48:22.173025 10016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34943
I20260927 22:48:22.173324 10016 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:22.174304 10016 heartbeater.cc:511] Master 127.0.97.62:34943 requested a full tablet report, sending...
I20260927 22:48:22.176003 9699 ts_manager.cc:194] Registered new tserver with Master: c212303dfc304054872b2ef61e48d398 (127.0.97.2:32901)
I20260927 22:48:22.177074 9699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45873
W20260927 22:48:22.493230 10019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:22.493484 10019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:22.493603 10019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:22.500689 10019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:22.500777 10019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:22.500906 10019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:48:22.509552 10019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:22.510696 10019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:22.512444 10019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:22.524333 10025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:22.524365 10024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:22.524829 10027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:22.525427 10019 server_base.cc:1034] running on GCE node
I20260927 22:48:22.526050 10019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:22.526944 10019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:22.528157 10019 hybrid_clock.cc:648] HybridClock initialized: now 1790549302528097 us; error 58 us; skew 500 ppm
I20260927 22:48:22.528448 10019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:22.530587 10019 webserver.cc:460] Webserver started at http://127.0.97.3:37027/ using document root <none> and password file <none>
I20260927 22:48:22.531235 10019 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:22.531329 10019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:22.531659 10019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:22.534330 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "bc7b1b7b9f28480386de0fdc5a2a1b1c"
format_stamp: "Formatted at 2026-09-27 22:48:22 on dist-test-slave-kr25"
server_key: "d46ee5750914913b1dcd348bcc96d3c3"
server_key_iv: "02f6d0da271fb69c33722146347e35f4"
server_key_version: "encryptionkey@0"
I20260927 22:48:22.535048 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bc7b1b7b9f28480386de0fdc5a2a1b1c"
format_stamp: "Formatted at 2026-09-27 22:48:22 on dist-test-slave-kr25"
server_key: "d46ee5750914913b1dcd348bcc96d3c3"
server_key_iv: "02f6d0da271fb69c33722146347e35f4"
server_key_version: "encryptionkey@0"
I20260927 22:48:22.539793 10019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:48:22.542814 10033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:22.544107 10019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:48:22.544248 10019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "bc7b1b7b9f28480386de0fdc5a2a1b1c"
format_stamp: "Formatted at 2026-09-27 22:48:22 on dist-test-slave-kr25"
server_key: "d46ee5750914913b1dcd348bcc96d3c3"
server_key_iv: "02f6d0da271fb69c33722146347e35f4"
server_key_version: "encryptionkey@0"
I20260927 22:48:22.544423 10019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:22.563979 10019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:22.564755 10019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:22.564996 10019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:22.565924 10019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:22.567287 10019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:22.567368 10019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:22.567459 10019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:22.567526 10019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:48:22.595500 10019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38737
I20260927 22:48:22.595569 10145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38737 every 8 connection(s)
I20260927 22:48:22.596750 10019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:48:22.599722 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10019
I20260927 22:48:22.599882 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:48:22.600410 388 external_mini_cluster.cc:1428] Setting key fe44cf5f233ebb1137e71ea1e6bcf9e9
I20260927 22:48:22.608601 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:22.599003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48507 (local address 127.0.97.62:34943)
0927 22:48:22.599167 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:22.599171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:22.600750 (+ 1579us) server_negotiation.cc:407] Connection header received
0927 22:48:22.602597 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:22.602602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:22.602681 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:22.602761 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:22.603855 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.604586 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.606109 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.606508 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.606611 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:22.607857 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:22.607862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:22.607866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:22.607882 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:22.607927 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:22.607942 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:22.608024 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:22.608211 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 22:48:22.608297 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:48:22.609489 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:22.599576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34943 (local address 127.0.97.3:48507)
0927 22:48:22.600601 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:22.600647 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:22.602355 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:22.603001 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:22.603012 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:22.603129 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:22.603709 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:22.603726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.604722 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.604726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:22.605980 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:22.605991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.606701 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.606708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:22.606954 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:22.607571 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:48:22.607573 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:22.607609 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:48:22.607698 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:22.608017 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:22.608027 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:22.608034 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:48:22.608175 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:48:22.608543 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":321,"threads_started":1}
I20260927 22:48:22.610657 10146 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34943
I20260927 22:48:22.610884 10146 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:22.611598 10146 heartbeater.cc:511] Master 127.0.97.62:34943 requested a full tablet report, sending...
I20260927 22:48:22.612812 9699 ts_manager.cc:194] Registered new tserver with Master: bc7b1b7b9f28480386de0fdc5a2a1b1c (127.0.97.3:38737)
I20260927 22:48:22.613586 9699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:48507
I20260927 22:48:22.615218 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:48:22.620726 9887 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:22.619181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.0.97.62:34943)
0927 22:48:22.619317 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:22.619322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:22.619806 (+ 484us) server_negotiation.cc:407] Connection header received
0927 22:48:22.619969 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:22.619974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:22.620054 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:22.620147 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:22.620521 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33434: BlockingRecv error: recv got EOF from 127.0.0.1:33434 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 22:48:22.621234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9758
I20260927 22:48:22.628214 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:22.622335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.0.97.1:46645)
0927 22:48:22.622862 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:22.622867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:22.622890 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:48:22.622939 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:22.622944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:22.623126 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:22.623235 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:22.623605 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.624519 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.625640 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.626126 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.626215 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:22.627048 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:22.627055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:22.627066 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:22.627605 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:22.627646 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:22.627664 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:22.627740 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:22.627847 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 22:48:22.627937 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
I20260927 22:48:22.629230 9880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:22.709664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 9758
I20260927 22:48:22.723006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9889
I20260927 22:48:22.730166 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:22.723922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.97.2:32901)
0927 22:48:22.724454 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:22.724461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:22.724486 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:48:22.724551 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:22.724557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:22.724761 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:22.724882 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:22.725203 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.726196 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.727280 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.727933 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.728030 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:22.728896 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:22.728903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:22.728916 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:22.729453 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:22.729505 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:22.729523 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:22.729619 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:22.729727 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 22:48:22.729838 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
I20260927 22:48:22.731244 10011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:22.809499 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 9889
I20260927 22:48:22.822059 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10019
I20260927 22:48:22.829640 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:22.822913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.0.97.3:38737)
0927 22:48:22.823552 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:22.823559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:22.823587 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:48:22.823645 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:22.823651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:22.823861 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:22.824011 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:22.824448 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.825431 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.826610 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:22.827146 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:22.827252 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:22.828330 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:22.828339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:22.828352 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:22.828894 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:22.828935 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:22.828953 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:22.829056 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:22.829207 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 22:48:22.829323 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":312,"threads_started":1}
I20260927 22:48:22.830772 10141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:22.910411 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10019
I20260927 22:48:22.923393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 9668
I20260927 22:48:22.924566 9729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:23.022279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 9668
2026-09-27T22:48:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2309 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T22:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:48:23Z Disabled control of system clock
I20260927 22:48:23.084023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39905
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:48:23.387789 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:23.388067 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:23.388186 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:23.395352 10165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:48:23.395488 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:23.395556 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:23.395633 10165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:48:23.395696 10165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:48:23.404960 10165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:23.406199 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:23.407989 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:23.418942 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:23.418917 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:23.418968 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:23.419668 10165 server_base.cc:1034] running on GCE node
I20260927 22:48:23.420368 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:23.421658 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:23.422873 10165 hybrid_clock.cc:648] HybridClock initialized: now 1790549303422816 us; error 42 us; skew 500 ppm
I20260927 22:48:23.423192 10165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:23.425439 10165 webserver.cc:460] Webserver started at http://127.0.97.62:35419/ using document root <none> and password file <none>
I20260927 22:48:23.426065 10165 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:23.426149 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:23.426429 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:23.429132 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "4f8dc717dd294936b0897860b1c67a78"
format_stamp: "Formatted at 2026-09-27 22:48:23 on dist-test-slave-kr25"
server_key: "084ce255f1b836d79edf7439695f2a8b"
server_key_iv: "2c1e5af74b2261dee6a5685a77a4bd8b"
server_key_version: "encryptionkey@0"
I20260927 22:48:23.429862 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "4f8dc717dd294936b0897860b1c67a78"
format_stamp: "Formatted at 2026-09-27 22:48:23 on dist-test-slave-kr25"
server_key: "084ce255f1b836d79edf7439695f2a8b"
server_key_iv: "2c1e5af74b2261dee6a5685a77a4bd8b"
server_key_version: "encryptionkey@0"
I20260927 22:48:23.434408 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 22:48:23.437693 10179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:23.438902 10165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:48:23.439090 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "4f8dc717dd294936b0897860b1c67a78"
format_stamp: "Formatted at 2026-09-27 22:48:23 on dist-test-slave-kr25"
server_key: "084ce255f1b836d79edf7439695f2a8b"
server_key_iv: "2c1e5af74b2261dee6a5685a77a4bd8b"
server_key_version: "encryptionkey@0"
I20260927 22:48:23.439265 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:23.444541 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:23.445236 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:23.445487 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:23.461445 10165 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39905
I20260927 22:48:23.461463 10230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39905 every 8 connection(s)
I20260927 22:48:23.462644 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:48:23.466216 10231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:48:23.471803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10165
I20260927 22:48:23.471968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:48:23.472537 388 external_mini_cluster.cc:1428] Setting key 2266c87fdb921cfdb4f55e13437500a1
I20260927 22:48:23.476298 10231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78: Bootstrap starting.
I20260927 22:48:23.479425 10231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78: Neither blocks nor log segments found. Creating new log.
I20260927 22:48:23.480274 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:23.474395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.97.62:39905)
0927 22:48:23.475262 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:23.475272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:23.475312 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:48:23.476921 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:23.476942 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:23.477338 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:23.477684 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:23.478682 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:23.478688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:23.478702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:23.478715 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:23.478778 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:23.478794 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:23.478947 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:23.479079 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:48:23.479310 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":272,"threads_started":1}
I20260927 22:48:23.480538 10231 log.cc:824] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78: Log is configured to *not* fsync() on all Append() calls
I20260927 22:48:23.483309 10231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78: No bootstrap required, opened a new log
I20260927 22:48:23.490195 10231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8dc717dd294936b0897860b1c67a78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39905 } }
I20260927 22:48:23.490520 10231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:48:23.490590 10231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8dc717dd294936b0897860b1c67a78, State: Initialized, Role: FOLLOWER
I20260927 22:48:23.491096 10231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8dc717dd294936b0897860b1c67a78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39905 } }
I20260927 22:48:23.491288 10231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:48:23.491376 10231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:48:23.491541 10231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:48:23.493556 10231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8dc717dd294936b0897860b1c67a78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39905 } }
I20260927 22:48:23.494091 10231 leader_election.cc:302] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f8dc717dd294936b0897860b1c67a78; no voters:
I20260927 22:48:23.494640 10231 leader_election.cc:288] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:48:23.494728 10236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:48:23.495383 10236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8dc717dd294936b0897860b1c67a78, State: Running, Role: LEADER
I20260927 22:48:23.495867 10236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8dc717dd294936b0897860b1c67a78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39905 } }
I20260927 22:48:23.496526 10231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:48:23.498356 10238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f8dc717dd294936b0897860b1c67a78. Latest consensus state: current_term: 1 leader_uuid: "4f8dc717dd294936b0897860b1c67a78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8dc717dd294936b0897860b1c67a78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39905 } } }
I20260927 22:48:23.498380 10237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f8dc717dd294936b0897860b1c67a78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8dc717dd294936b0897860b1c67a78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39905 } } }
I20260927 22:48:23.498742 10238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:23.498739 10237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78 [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:23.504688 10242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:48:23.509392 10242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:48:23.516940 10242 catalog_manager.cc:1347] Generated new cluster ID: 5f16ad156ee74c76b2b3212af3be9d97
I20260927 22:48:23.517064 10242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:48:23.526558 10242 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:48:23.527055 10242 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:48:23.545002 10242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f8dc717dd294936b0897860b1c67a78: Generated new TSK 0
I20260927 22:48:23.546072 10242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:48:23.553094 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39905
--builtin_ntp_servers=127.0.97.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:48:23.865458 10255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:23.865715 10255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:23.865800 10255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:23.872845 10255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:23.872941 10255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:23.873054 10255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:48:23.881832 10255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:23.883001 10255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:23.884694 10255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:23.896694 10260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:23.896721 10261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:23.896654 10263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:23.897418 10255 server_base.cc:1034] running on GCE node
I20260927 22:48:23.898031 10255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:23.898895 10255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:23.900085 10255 hybrid_clock.cc:648] HybridClock initialized: now 1790549303900027 us; error 46 us; skew 500 ppm
I20260927 22:48:23.900394 10255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:23.902487 10255 webserver.cc:460] Webserver started at http://127.0.97.1:34263/ using document root <none> and password file <none>
I20260927 22:48:23.903108 10255 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:23.903204 10255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:23.903509 10255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:23.906143 10255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "a8e3d3ae704d4a7c81c214770b7e81ca"
format_stamp: "Formatted at 2026-09-27 22:48:23 on dist-test-slave-kr25"
server_key: "ea581940de1df0d18e8ff52101a06ad6"
server_key_iv: "228a56d4bda0e7336c71ffa42de26a80"
server_key_version: "encryptionkey@0"
I20260927 22:48:23.906860 10255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a8e3d3ae704d4a7c81c214770b7e81ca"
format_stamp: "Formatted at 2026-09-27 22:48:23 on dist-test-slave-kr25"
server_key: "ea581940de1df0d18e8ff52101a06ad6"
server_key_iv: "228a56d4bda0e7336c71ffa42de26a80"
server_key_version: "encryptionkey@0"
I20260927 22:48:23.911475 10255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:48:23.914561 10269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:23.915802 10255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260927 22:48:23.915939 10255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "a8e3d3ae704d4a7c81c214770b7e81ca"
format_stamp: "Formatted at 2026-09-27 22:48:23 on dist-test-slave-kr25"
server_key: "ea581940de1df0d18e8ff52101a06ad6"
server_key_iv: "228a56d4bda0e7336c71ffa42de26a80"
server_key_version: "encryptionkey@0"
I20260927 22:48:23.916131 10255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:23.921381 10255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:23.922086 10255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:23.922315 10255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:23.923178 10255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:23.924494 10255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:23.924594 10255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:23.924697 10255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:23.924763 10255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:23.953815 10255 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36963
I20260927 22:48:23.953866 10381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36963 every 8 connection(s)
I20260927 22:48:23.955070 10255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:48:23.962872 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:23.957629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53155 (local address 127.0.97.62:39905)
0927 22:48:23.957803 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:23.957808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:23.959145 (+ 1337us) server_negotiation.cc:407] Connection header received
0927 22:48:23.960913 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:23.960917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:23.960970 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:23.961064 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:23.962358 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:23.962361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:23.962364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:23.962377 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:23.962401 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:23.962411 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:23.962496 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:23.962640 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 22:48:23.962684 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:48:23.963178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10255
I20260927 22:48:23.963407 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:48:23.964262 388 external_mini_cluster.cc:1428] Setting key c072336af437dafba4a5df0b2b8a40fc
I20260927 22:48:23.964349 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:23.958076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39905 (local address 127.0.97.1:53155)
0927 22:48:23.958997 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:23.959040 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:23.960646 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:23.961277 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:23.961285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:23.961379 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:23.962083 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:48:23.962085 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:23.962124 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:48:23.962212 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:23.962480 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:23.962486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:23.962492 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:48:23.962744 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:48:23.963059 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":267,"threads_started":1}
I20260927 22:48:23.965785 10382 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39905
I20260927 22:48:23.966084 10382 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:23.966671 10382 heartbeater.cc:511] Master 127.0.97.62:39905 requested a full tablet report, sending...
I20260927 22:48:23.966938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39905
--builtin_ntp_servers=127.0.97.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:48:23.968767 10196 ts_manager.cc:194] Registered new tserver with Master: a8e3d3ae704d4a7c81c214770b7e81ca (127.0.97.1:36963)
W20260927 22:48:24.273726 10385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:24.273999 10385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:24.274117 10385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:24.281131 10385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:24.281231 10385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:24.281371 10385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:48:24.290223 10385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:24.291461 10385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:24.293162 10385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:24.305334 10390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:24.305377 10393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:24.305405 10391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:24.306159 10385 server_base.cc:1034] running on GCE node
I20260927 22:48:24.306674 10385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:24.307538 10385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:24.308724 10385 hybrid_clock.cc:648] HybridClock initialized: now 1790549304308675 us; error 43 us; skew 500 ppm
I20260927 22:48:24.309001 10385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:24.310952 10385 webserver.cc:460] Webserver started at http://127.0.97.2:45881/ using document root <none> and password file <none>
I20260927 22:48:24.311599 10385 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:24.311697 10385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:24.311978 10385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:24.316205 10385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "198f07c472a343df907863c8af6e4aa0"
format_stamp: "Formatted at 2026-09-27 22:48:24 on dist-test-slave-kr25"
server_key: "b227d6a3cfc00254d7be89e7a913e262"
server_key_iv: "5e6b6dad023ca7c02ff46539c913b67a"
server_key_version: "encryptionkey@0"
I20260927 22:48:24.316931 10385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "198f07c472a343df907863c8af6e4aa0"
format_stamp: "Formatted at 2026-09-27 22:48:24 on dist-test-slave-kr25"
server_key: "b227d6a3cfc00254d7be89e7a913e262"
server_key_iv: "5e6b6dad023ca7c02ff46539c913b67a"
server_key_version: "encryptionkey@0"
I20260927 22:48:24.321763 10385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:48:24.325138 10399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:24.326417 10385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:48:24.326560 10385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "198f07c472a343df907863c8af6e4aa0"
format_stamp: "Formatted at 2026-09-27 22:48:24 on dist-test-slave-kr25"
server_key: "b227d6a3cfc00254d7be89e7a913e262"
server_key_iv: "5e6b6dad023ca7c02ff46539c913b67a"
server_key_version: "encryptionkey@0"
I20260927 22:48:24.326756 10385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:24.331939 10385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:24.332650 10385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:24.332876 10385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:24.333709 10385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:24.334977 10385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:24.335074 10385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:24.335175 10385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:24.335242 10385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:24.364145 10385 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40971
I20260927 22:48:24.364219 10511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40971 every 8 connection(s)
I20260927 22:48:24.365410 10385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:48:24.366042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10385
I20260927 22:48:24.366232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:48:24.366834 388 external_mini_cluster.cc:1428] Setting key 980dfc89e5ea287efd94a3cd8339c848
I20260927 22:48:24.369679 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39905
--builtin_ntp_servers=127.0.97.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:48:24.374027 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:24.367901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46873 (local address 127.0.97.62:39905)
0927 22:48:24.368088 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:24.368094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:24.369895 (+ 1801us) server_negotiation.cc:407] Connection header received
0927 22:48:24.372011 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:24.372015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:24.372066 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:24.372148 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:24.373443 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:24.373447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:24.373450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:24.373465 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:24.373494 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:24.373505 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:24.373610 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:24.373777 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 22:48:24.373823 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:48:24.375150 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:24.368522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39905 (local address 127.0.97.2:46873)
0927 22:48:24.369621 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:24.369681 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:24.371747 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:24.372376 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:24.372387 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:24.372497 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:24.373129 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:48:24.373132 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:24.373185 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:48:24.373282 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:24.373580 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:24.373587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:24.373593 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:48:24.373856 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:48:24.374094 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":284,"threads_started":1}
I20260927 22:48:24.376433 10512 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39905
I20260927 22:48:24.376700 10512 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:24.377230 10512 heartbeater.cc:511] Master 127.0.97.62:39905 requested a full tablet report, sending...
I20260927 22:48:24.378466 10196 ts_manager.cc:194] Registered new tserver with Master: 198f07c472a343df907863c8af6e4aa0 (127.0.97.2:40971)
W20260927 22:48:24.681450 10515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:24.681725 10515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:24.681860 10515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:24.689009 10515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:24.689112 10515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:24.689257 10515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:48:24.698136 10515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:24.699291 10515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:24.701122 10515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:24.712997 10523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:24.713061 10520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:24.712955 10521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:24.713683 10515 server_base.cc:1034] running on GCE node
I20260927 22:48:24.714259 10515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:24.715087 10515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:24.716269 10515 hybrid_clock.cc:648] HybridClock initialized: now 1790549304716232 us; error 38 us; skew 500 ppm
I20260927 22:48:24.716548 10515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:48:24.718645 10515 webserver.cc:460] Webserver started at http://127.0.97.3:33811/ using document root <none> and password file <none>
I20260927 22:48:24.719292 10515 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:24.719372 10515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:24.719662 10515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:24.722360 10515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "e6036f4df63f409cb27d747aaf985cde"
format_stamp: "Formatted at 2026-09-27 22:48:24 on dist-test-slave-kr25"
server_key: "8de74f4f46a9b1d4df9928f8162ed607"
server_key_iv: "2c7d1f0858727ec03ab13b1e6b6cdcbd"
server_key_version: "encryptionkey@0"
I20260927 22:48:24.723069 10515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e6036f4df63f409cb27d747aaf985cde"
format_stamp: "Formatted at 2026-09-27 22:48:24 on dist-test-slave-kr25"
server_key: "8de74f4f46a9b1d4df9928f8162ed607"
server_key_iv: "2c7d1f0858727ec03ab13b1e6b6cdcbd"
server_key_version: "encryptionkey@0"
I20260927 22:48:24.727852 10515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260927 22:48:24.731096 10529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:24.732367 10515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:48:24.732496 10515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "e6036f4df63f409cb27d747aaf985cde"
format_stamp: "Formatted at 2026-09-27 22:48:24 on dist-test-slave-kr25"
server_key: "8de74f4f46a9b1d4df9928f8162ed607"
server_key_iv: "2c7d1f0858727ec03ab13b1e6b6cdcbd"
server_key_version: "encryptionkey@0"
I20260927 22:48:24.732693 10515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:24.737823 10515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:24.738508 10515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:24.738734 10515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:24.739557 10515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:24.740868 10515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:24.740954 10515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:24.741056 10515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:24.741120 10515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:24.770083 10515 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32939
I20260927 22:48:24.770131 10641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32939 every 8 connection(s)
I20260927 22:48:24.771318 10515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:48:24.778854 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:24.773867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34221 (local address 127.0.97.62:39905)
0927 22:48:24.774068 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:24.774074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:24.775491 (+ 1417us) server_negotiation.cc:407] Connection header received
0927 22:48:24.777122 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:24.777127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:24.777176 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:24.777263 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:24.778343 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:24.778347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:24.778351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:24.778364 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:24.778392 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:24.778402 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:24.778485 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:24.778616 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:48:24.778660 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:48:24.779747 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:24.774448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39905 (local address 127.0.97.3:34221)
0927 22:48:24.775333 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:24.775377 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:24.776874 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:24.777434 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:24.777442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:24.777530 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:24.778088 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:48:24.778091 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:24.778126 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:48:24.778205 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:24.778473 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:24.778478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:24.778483 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:48:24.778678 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 22:48:24.778854 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":276,"threads_started":1}
I20260927 22:48:24.781065 10642 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39905
I20260927 22:48:24.781276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10515
I20260927 22:48:24.781320 10642 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:24.781461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:48:24.781848 10642 heartbeater.cc:511] Master 127.0.97.62:39905 requested a full tablet report, sending...
I20260927 22:48:24.781860 388 external_mini_cluster.cc:1428] Setting key a7cd65656c839bfef5b302d23c04fc2d
I20260927 22:48:24.783298 10195 ts_manager.cc:194] Registered new tserver with Master: e6036f4df63f409cb27d747aaf985cde (127.0.97.3:32939)
I20260927 22:48:24.784710 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:48:24.790153 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:48:24.788881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.0.97.62:39905)
0927 22:48:24.789003 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:24.789007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:24.789468 (+ 461us) server_negotiation.cc:407] Connection header received
0927 22:48:24.789572 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:24.789575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:24.789619 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:24.789686 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:24.789931 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45178: BlockingRecv error: recv got EOF from 127.0.0.1:45178 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:48:24.791147 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10255
I20260927 22:48:24.794018 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:24.791937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.0.97.1:36963)
0927 22:48:24.792403 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:24.792408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:24.792431 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:48:24.792489 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:24.792495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:24.792668 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:24.792779 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:24.793482 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:24.793489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:24.793498 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:24.793514 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:24.793549 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:24.793563 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:24.793656 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:24.793728 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 22:48:24.793802 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260927 22:48:24.795073 10377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:24.873591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10255
I20260927 22:48:24.886411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10385
I20260927 22:48:24.889729 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:24.887260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.97.2:40971)
0927 22:48:24.887772 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:24.887779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:24.887805 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:48:24.887865 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:24.887871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:24.888106 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:24.888231 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:24.889125 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:24.889131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:24.889144 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:24.889162 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:24.889205 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:24.889219 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:24.889320 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:24.889427 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 22:48:24.889503 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":271,"threads_started":1}
I20260927 22:48:24.890868 10507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:24.969637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10385
I20260927 22:48:24.982432 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10515
I20260927 22:48:24.986512 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:24.983407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.0.97.3:32939)
0927 22:48:24.984090 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:24.984097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:24.984129 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:48:24.984195 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:24.984201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:24.984412 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:24.984584 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:24.985737 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:24.985743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:24.985755 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:48:24.985773 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:24.985812 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:48:24.985827 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:24.985991 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:24.986101 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 22:48:24.986201 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":340,"threads_started":1}
I20260927 22:48:24.987782 10637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:25.066663 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10515
I20260927 22:48:25.078830 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10165
I20260927 22:48:25.079972 10226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:25.176585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10165
2026-09-27T22:48:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2153 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:48:25 dist-test-slave-kr25 krb5kdc[10660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:48:25 dist-test-slave-kr25 krb5kdc[10660](info): set up 2 sockets
Sep 27 22:48:25 dist-test-slave-kr25 krb5kdc[10660](info): commencing operation
krb5kdc: starting...
W20260927 22:48:26.179957 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:48:26 dist-test-slave-kr25 krb5kdc[10660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:48:26.388461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34343
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:26.695508 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:26.695777 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:26.695871 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:26.702903 10673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:48:26.703006 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:26.703059 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:26.703114 10673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:48:26.703155 10673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:48:26.712419 10673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:26.713546 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:26.714720 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:26.725296 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:26.725258 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:26.725323 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:26.726104 10673 server_base.cc:1034] running on GCE node
I20260927 22:48:26.726832 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:26.728207 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:26.729414 10673 hybrid_clock.cc:648] HybridClock initialized: now 1790549306729369 us; error 36 us; skew 500 ppm
I20260927 22:48:26.729733 10673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:26 dist-test-slave-kr25 krb5kdc[10660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:26.732632 10673 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:48:26.733992 10673 webserver.cc:460] Webserver started at http://127.0.97.62:34485/ using document root <none> and password file <none>
I20260927 22:48:26.734607 10673 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:26.734697 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:26.734926 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:26.737665 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "fce0ed592cac47b58c29800049647344"
format_stamp: "Formatted at 2026-09-27 22:48:26 on dist-test-slave-kr25"
server_key: "27fa7abfe5a1642527d9c12c5bca03c0"
server_key_iv: "9f5b928dacdcf204cf9d53b37c8edf0d"
server_key_version: "encryptionkey@0"
I20260927 22:48:26.738334 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "fce0ed592cac47b58c29800049647344"
format_stamp: "Formatted at 2026-09-27 22:48:26 on dist-test-slave-kr25"
server_key: "27fa7abfe5a1642527d9c12c5bca03c0"
server_key_iv: "9f5b928dacdcf204cf9d53b37c8edf0d"
server_key_version: "encryptionkey@0"
I20260927 22:48:26.742930 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:48:26.746070 10688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:26.747247 10673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:48:26.747373 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "fce0ed592cac47b58c29800049647344"
format_stamp: "Formatted at 2026-09-27 22:48:26 on dist-test-slave-kr25"
server_key: "27fa7abfe5a1642527d9c12c5bca03c0"
server_key_iv: "9f5b928dacdcf204cf9d53b37c8edf0d"
server_key_version: "encryptionkey@0"
I20260927 22:48:26.747555 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:26.758688 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:26.762537 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:26.762750 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:26.778972 10673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34343
I20260927 22:48:26.779043 10739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34343 every 8 connection(s)
I20260927 22:48:26.780253 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:48:26.783741 10740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:48:26.787822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10673
I20260927 22:48:26.788012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:48:26.788589 388 external_mini_cluster.cc:1428] Setting key 0dd05095cf8b4e0f0df3eb0671e029ea
I20260927 22:48:26.795118 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344: Bootstrap starting.
I20260927 22:48:26.798601 10740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344: Neither blocks nor log segments found. Creating new log.
I20260927 22:48:26.799710 10740 log.cc:824] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344: Log is configured to *not* fsync() on all Append() calls
I20260927 22:48:26.802493 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344: No bootstrap required, opened a new log
Sep 27 22:48:26 dist-test-slave-kr25 krb5kdc[10660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:26.812112 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:26.790429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.0.97.62:34343)
0927 22:48:26.791220 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:26.791235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:26.791280 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:48:26.793636 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:26.793664 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:26.794155 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:26.794597 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:26.795937 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:26.797838 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:26.799123 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:26.800135 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:26.800345 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:26.804039 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:26.804049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:26.804068 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:26.804121 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:26.807900 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:26.808519 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:26.808527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:26.808536 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:26.808734 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:26.809030 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:26.809035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:26.809038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:26.809673 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:26.809949 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:26.810278 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:48:26.810712 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":310,"threads_started":1}
I20260927 22:48:26.812335 10740 raft_consensus.cc:348] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce0ed592cac47b58c29800049647344" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34343 } }
I20260927 22:48:26.812712 10740 raft_consensus.cc:374] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:48:26.812801 10740 raft_consensus.cc:729] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce0ed592cac47b58c29800049647344, State: Initialized, Role: FOLLOWER
I20260927 22:48:26.813362 10740 consensus_queue.cc:260] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce0ed592cac47b58c29800049647344" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34343 } }
I20260927 22:48:26.813635 10740 raft_consensus.cc:388] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:48:26.813813 10740 raft_consensus.cc:482] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:48:26.813992 10740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:48:26.816318 10740 raft_consensus.cc:504] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce0ed592cac47b58c29800049647344" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34343 } }
I20260927 22:48:26.816730 10740 leader_election.cc:302] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fce0ed592cac47b58c29800049647344; no voters:
I20260927 22:48:26.817232 10740 leader_election.cc:288] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:48:26.817348 10745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:48:26.817943 10745 raft_consensus.cc:686] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [term 1 LEADER]: Becoming Leader. State: Replica: fce0ed592cac47b58c29800049647344, State: Running, Role: LEADER
I20260927 22:48:26.818449 10745 consensus_queue.cc:237] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce0ed592cac47b58c29800049647344" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34343 } }
I20260927 22:48:26.819010 10740 sys_catalog.cc:564] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:48:26.821046 10746 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fce0ed592cac47b58c29800049647344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce0ed592cac47b58c29800049647344" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34343 } } }
I20260927 22:48:26.821408 10746 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:26.821013 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce0ed592cac47b58c29800049647344. Latest consensus state: current_term: 1 leader_uuid: "fce0ed592cac47b58c29800049647344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce0ed592cac47b58c29800049647344" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34343 } } }
I20260927 22:48:26.821749 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344 [sys.catalog]: This master's current role is: LEADER
I20260927 22:48:26.822211 10753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:48:26.827553 10753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:48:26.835306 10753 catalog_manager.cc:1347] Generated new cluster ID: a05c237660534ddb81e052a5f65bc2d8
I20260927 22:48:26.835402 10753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:48:26.853263 10753 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:48:26.855031 10753 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:48:26.869014 10753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344: Generated new TSK 0
I20260927 22:48:26.869868 10753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:48:26.927747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34343
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:27.236804 10766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:27.237083 10766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:27.237226 10766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:27.244210 10766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:27.244328 10766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:27.244483 10766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:48:27.254185 10766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:27.255386 10766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:27.256654 10766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:27.268464 10771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:27.268455 10774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:27.268499 10772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:27.269546 10766 server_base.cc:1034] running on GCE node
I20260927 22:48:27.270066 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:27.270881 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:27.272078 10766 hybrid_clock.cc:648] HybridClock initialized: now 1790549307272002 us; error 60 us; skew 500 ppm
I20260927 22:48:27.272352 10766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:27 dist-test-slave-kr25 krb5kdc[10660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:27.278102 10766 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:48:27.279387 10766 webserver.cc:460] Webserver started at http://127.0.97.1:39363/ using document root <none> and password file <none>
I20260927 22:48:27.280090 10766 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:27.280203 10766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:27.280464 10766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:27.283185 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "5eada93b095b4ff588cf9dc47931cc21"
format_stamp: "Formatted at 2026-09-27 22:48:27 on dist-test-slave-kr25"
server_key: "e8afbd22b9cf02b49e2038de325a78b0"
server_key_iv: "19451a86b7dc33db47cfc0ecc3d4f1cc"
server_key_version: "encryptionkey@0"
I20260927 22:48:27.283931 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5eada93b095b4ff588cf9dc47931cc21"
format_stamp: "Formatted at 2026-09-27 22:48:27 on dist-test-slave-kr25"
server_key: "e8afbd22b9cf02b49e2038de325a78b0"
server_key_iv: "19451a86b7dc33db47cfc0ecc3d4f1cc"
server_key_version: "encryptionkey@0"
I20260927 22:48:27.288687 10766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:48:27.291837 10781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:27.293100 10766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:48:27.293257 10766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "5eada93b095b4ff588cf9dc47931cc21"
format_stamp: "Formatted at 2026-09-27 22:48:27 on dist-test-slave-kr25"
server_key: "e8afbd22b9cf02b49e2038de325a78b0"
server_key_iv: "19451a86b7dc33db47cfc0ecc3d4f1cc"
server_key_version: "encryptionkey@0"
I20260927 22:48:27.293434 10766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:27.312457 10766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:27.316813 10766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:27.317119 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:27.318066 10766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:27.319558 10766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:27.319644 10766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:27.319808 10766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:27.319865 10766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:27.349088 10766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36155
I20260927 22:48:27.349164 10893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36155 every 8 connection(s)
I20260927 22:48:27.350283 10766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:48:27.360383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10766
I20260927 22:48:27.360618 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:48:27.361219 388 external_mini_cluster.cc:1428] Setting key c285970893e5289eb40a12f41870529a
Sep 27 22:48:27 dist-test-slave-kr25 krb5kdc[10660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:27.369606 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:27.352713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59007 (local address 127.0.97.62:34343)
0927 22:48:27.353137 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:27.353142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:27.354164 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 22:48:27.355940 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:27.355944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:27.356020 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:27.356115 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:27.357570 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.358298 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.359906 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.360529 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.360667 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:27.364247 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:27.364252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:27.364255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:27.364286 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:27.366609 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:27.367291 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:27.367295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:27.367298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:27.367407 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:27.367842 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:27.367846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:27.367848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:27.368132 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:27.368255 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:27.369008 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 22:48:27.369252 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":188,"threads_started":1}
I20260927 22:48:27.370548 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:27.353274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34343 (local address 127.0.97.1:59007)
0927 22:48:27.353962 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:27.354052 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:27.355686 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:27.356305 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:27.356314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:27.356804 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:27.357399 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:27.357415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.358448 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.358452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:27.359763 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:27.359774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.360744 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.360752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:27.360946 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:27.361600 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:27.361625 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:27.364018 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:27.366733 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:27.366742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:27.366760 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:27.367170 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:27.367526 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:27.367531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:27.367535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:27.367713 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:27.368247 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:27.368256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:27.368637 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 22:48:27.368913 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 22:48:27.369459 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":253,"threads_started":1}
I20260927 22:48:27.372042 10894 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34343
I20260927 22:48:27.372334 10894 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:27.373088 10894 heartbeater.cc:511] Master 127.0.97.62:34343 requested a full tablet report, sending...
I20260927 22:48:27.375030 10705 ts_manager.cc:194] Registered new tserver with Master: 5eada93b095b4ff588cf9dc47931cc21 (127.0.97.1:36155)
I20260927 22:48:27.376439 10705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59007
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:48:27.407335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34343
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:27.714006 10900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:27.714248 10900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:27.714336 10900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:27.721312 10900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:27.721411 10900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:27.721519 10900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:48:27.730801 10900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:27.731987 10900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:27.733163 10900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:27.745203 10905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:27.745180 10908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:27.745224 10906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:27.745895 10900 server_base.cc:1034] running on GCE node
I20260927 22:48:27.746409 10900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:27.747233 10900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:27.748426 10900 hybrid_clock.cc:648] HybridClock initialized: now 1790549307748382 us; error 32 us; skew 500 ppm
I20260927 22:48:27.748708 10900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:27 dist-test-slave-kr25 krb5kdc[10660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:27.751523 10900 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:48:27.752771 10900 webserver.cc:460] Webserver started at http://127.0.97.2:34817/ using document root <none> and password file <none>
I20260927 22:48:27.753355 10900 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:27.753448 10900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:27.753712 10900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:27.756500 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "2f7042458bda4f74bf28cab066a493a4"
format_stamp: "Formatted at 2026-09-27 22:48:27 on dist-test-slave-kr25"
server_key: "238a1961d8f3ad28e0bf9ecbe936c779"
server_key_iv: "7071c3daa799dffea91a360cfb5842c8"
server_key_version: "encryptionkey@0"
I20260927 22:48:27.757184 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2f7042458bda4f74bf28cab066a493a4"
format_stamp: "Formatted at 2026-09-27 22:48:27 on dist-test-slave-kr25"
server_key: "238a1961d8f3ad28e0bf9ecbe936c779"
server_key_iv: "7071c3daa799dffea91a360cfb5842c8"
server_key_version: "encryptionkey@0"
I20260927 22:48:27.761831 10900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:48:27.765012 10915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:27.766351 10900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:48:27.766535 10900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "2f7042458bda4f74bf28cab066a493a4"
format_stamp: "Formatted at 2026-09-27 22:48:27 on dist-test-slave-kr25"
server_key: "238a1961d8f3ad28e0bf9ecbe936c779"
server_key_iv: "7071c3daa799dffea91a360cfb5842c8"
server_key_version: "encryptionkey@0"
I20260927 22:48:27.766759 10900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:27.781684 10900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:27.785634 10900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:27.785905 10900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:27.786859 10900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:27.788213 10900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:27.788288 10900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:27.788403 10900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:27.788458 10900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:27.819142 10900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42791
I20260927 22:48:27.819178 11027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42791 every 8 connection(s)
I20260927 22:48:27.820528 10900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:48:27.827042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 10900
I20260927 22:48:27.827221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:48:27.827646 388 external_mini_cluster.cc:1428] Setting key 09a0334bf2d98702ca95b4e1c31ced53
Sep 27 22:48:27 dist-test-slave-kr25 krb5kdc[10660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:27.838721 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:27.822857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52889 (local address 127.0.97.62:34343)
0927 22:48:27.823043 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:27.823048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:27.824204 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 22:48:27.825959 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:27.825965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:27.826050 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:27.826134 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:27.827582 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.828406 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.829849 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.830238 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.830342 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:27.833548 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:27.833552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:27.833556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:27.833584 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:27.835979 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:27.836732 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:27.836736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:27.836738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:27.836849 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:27.837229 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:27.837231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:27.837233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:27.837445 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:27.837539 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:27.838322 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 22:48:27.838482 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:48:27.839903 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:27.823301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34343 (local address 127.0.97.2:52889)
0927 22:48:27.824047 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:27.824095 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:27.825714 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:27.826355 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:27.826364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:27.826834 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:27.827403 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:27.827423 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.828535 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.828540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:27.829726 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:27.829737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:27.830431 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:27.830438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:27.830658 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:27.831250 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:27.831271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:27.833363 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:27.836105 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:27.836114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:27.836128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:27.836636 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:27.836930 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:27.836935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:27.836938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:27.837144 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:27.837550 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:27.837557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:27.837912 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 22:48:27.838194 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:48:27.838662 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":305,"threads_started":1}
I20260927 22:48:27.841271 11028 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34343
I20260927 22:48:27.841553 11028 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:27.842242 11028 heartbeater.cc:511] Master 127.0.97.62:34343 requested a full tablet report, sending...
I20260927 22:48:27.843665 10705 ts_manager.cc:194] Registered new tserver with Master: 2f7042458bda4f74bf28cab066a493a4 (127.0.97.2:42791)
I20260927 22:48:27.844501 10705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52889
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:48:27.874096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34343
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:48:28.181022 11033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:48:28.181376 11033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:48:28.181586 11033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:48:28.188741 11033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:48:28.188862 11033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:48:28.189026 11033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:48:28.199004 11033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:48:28.200297 11033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:48:28.201586 11033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:48:28.214226 11038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:28.214259 11041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:48:28.214214 11039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:48:28.214967 11033 server_base.cc:1034] running on GCE node
I20260927 22:48:28.215569 11033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:48:28.216454 11033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:48:28.217653 11033 hybrid_clock.cc:648] HybridClock initialized: now 1790549308217590 us; error 52 us; skew 500 ppm
I20260927 22:48:28.217955 11033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:48:28 dist-test-slave-kr25 krb5kdc[10660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:48:28.221058 11033 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:48:28.222414 11033 webserver.cc:460] Webserver started at http://127.0.97.3:35245/ using document root <none> and password file <none>
I20260927 22:48:28.223057 11033 fs_manager.cc:359] Metadata directory not provided
I20260927 22:48:28.223158 11033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:48:28.223423 11033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:48:28.226218 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "5c985fbc25b34a10a76086b6cd7e3211"
format_stamp: "Formatted at 2026-09-27 22:48:28 on dist-test-slave-kr25"
server_key: "e9305d2fcba5e3a6355146ae3e42ccb0"
server_key_iv: "98dad5abb2f53fe3c5ebeec12a3f1760"
server_key_version: "encryptionkey@0"
I20260927 22:48:28.226902 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5c985fbc25b34a10a76086b6cd7e3211"
format_stamp: "Formatted at 2026-09-27 22:48:28 on dist-test-slave-kr25"
server_key: "e9305d2fcba5e3a6355146ae3e42ccb0"
server_key_iv: "98dad5abb2f53fe3c5ebeec12a3f1760"
server_key_version: "encryptionkey@0"
I20260927 22:48:28.231853 11033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:48:28.234982 11048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:28.236291 11033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:48:28.236433 11033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "5c985fbc25b34a10a76086b6cd7e3211"
format_stamp: "Formatted at 2026-09-27 22:48:28 on dist-test-slave-kr25"
server_key: "e9305d2fcba5e3a6355146ae3e42ccb0"
server_key_iv: "98dad5abb2f53fe3c5ebeec12a3f1760"
server_key_version: "encryptionkey@0"
I20260927 22:48:28.236604 11033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:48:28.263857 11033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:48:28.268141 11033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:48:28.268394 11033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:48:28.269294 11033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:48:28.270592 11033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:48:28.270668 11033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:28.270778 11033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:48:28.270828 11033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:48:28.300158 11033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44175
I20260927 22:48:28.300210 11160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44175 every 8 connection(s)
I20260927 22:48:28.301345 11033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:48:28.303308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 11033
I20260927 22:48:28.303506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:48:28.304008 388 external_mini_cluster.cc:1428] Setting key c31a7705e18fc98c1f7b6c841468e69a
Sep 27 22:48:28 dist-test-slave-kr25 krb5kdc[10660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:48:28.319789 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.303819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49777 (local address 127.0.97.62:34343)
0927 22:48:28.304012 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.304018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.305254 (+ 1236us) server_negotiation.cc:407] Connection header received
0927 22:48:28.307190 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.307196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.307272 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.307351 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:28.308758 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.309471 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.310903 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.311286 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.311387 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.314547 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:28.314552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:28.314555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:28.314586 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:28.317011 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.317676 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.317681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.317684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.317791 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.318172 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.318177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.318180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.318431 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:28.318552 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.319251 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.319482 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:48:28.320752 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.304352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34343 (local address 127.0.97.3:49777)
0927 22:48:28.305098 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:28.305144 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:28.306947 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:28.307622 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:28.307630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:28.308095 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:48:28.308615 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.308631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.309609 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.309614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.310780 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.310791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.311498 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.311513 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.311739 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.312354 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:48:28.312374 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:48:28.314360 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:48:28.317152 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:28.317160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:28.317175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:28.317571 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:28.317886 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:48:28.317891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:48:28.317894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:48:28.318067 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:48:28.318569 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:48:28.318577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:48:28.318875 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 22:48:28.319138 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:48:28.319639 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":300,"threads_started":1}
I20260927 22:48:28.322081 11161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34343
I20260927 22:48:28.322381 11161 heartbeater.cc:464] Registering TS with master...
I20260927 22:48:28.323145 11161 heartbeater.cc:511] Master 127.0.97.62:34343 requested a full tablet report, sending...
I20260927 22:48:28.324606 10705 ts_manager.cc:194] Registered new tserver with Master: 5c985fbc25b34a10a76086b6cd7e3211 (127.0.97.3:44175)
I20260927 22:48:28.325461 10705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49777
I20260927 22:48:28.331615 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:48:28.346736 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.335655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.97.62:34343)
0927 22:48:28.335829 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.335835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.336271 (+ 436us) server_negotiation.cc:407] Connection header received
0927 22:48:28.336402 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.336405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.336464 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.336542 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:28.337692 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.338358 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.339530 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.339957 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.340044 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.342191 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:28.342196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:28.342199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:28.342227 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:28.344631 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.345176 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.345180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.345182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.345289 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.345591 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.345594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.345597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.345876 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:28.346040 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.346308 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.346456 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:48:28.350400 10705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:48:28.357050 10705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:48:28.379118 10894 heartbeater.cc:503] Master 127.0.97.62:34343 was elected leader, sending a full tablet report...
I20260927 22:48:28.384595 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.368362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36155 (local address 127.0.0.1:36936)
0927 22:48:28.369201 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:28.369255 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:28.369466 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:28.376583 (+ 7117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:28.376590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:28.376655 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:48:28.377491 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.377502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.379313 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.379318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.381839 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.381849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.383581 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.383587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.383752 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.383803 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:48:28.383944 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:48:28.384134 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":833,"spinlock_wait_cycles":9472,"thread_start_us":7067,"threads_started":1}
I20260927 22:48:28.384871 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.376041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.0.97.2:42791)
0927 22:48:28.376507 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.376514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.377285 (+ 771us) server_negotiation.cc:407] Connection header received
0927 22:48:28.377564 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.377572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.377791 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.377939 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:48:28.378563 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.379935 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.382673 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.383640 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.383763 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.383934 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.384418 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.384577 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":241,"threads_started":1}
I20260927 22:48:28.385396 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.376551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42791 (local address 127.0.0.1:59014)
0927 22:48:28.377171 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:28.377204 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:28.377374 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:28.378128 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:28.378131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:28.378148 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:48:28.378406 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.378412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.380090 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.380094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.382545 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.382554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.384069 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.384075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.384220 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.384251 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:48:28.385015 (+ 764us) client_negotiation.cc:240] Negotiation successful
0927 22:48:28.385131 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":6,"thread_start_us":342,"threads_started":1}
I20260927 22:48:28.385392 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.368615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.0.97.1:36155)
0927 22:48:28.375997 (+ 7382us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.376005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.376029 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:48:28.376109 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.376115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.376349 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.377059 (+ 710us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:48:28.377681 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.379177 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.381969 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.383385 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.383544 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.383774 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.384932 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.385078 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7240,"thread_start_us":7086,"threads_started":1}
I20260927 22:48:28.385406 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.369026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.97.3:44175)
0927 22:48:28.375399 (+ 6373us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.375409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.375780 (+ 371us) server_negotiation.cc:407] Connection header received
0927 22:48:28.376080 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.376088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.376349 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.376542 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:48:28.377461 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.379078 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.382107 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.383388 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.383544 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.383774 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.384908 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.385067 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6180,"thread_start_us":6015,"threads_started":1}
I20260927 22:48:28.384778 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.368485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44175 (local address 127.0.0.1:44272)
0927 22:48:28.375672 (+ 7187us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:28.375696 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:28.375889 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:28.376592 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:28.376597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:28.376641 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:48:28.377249 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.377263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.379238 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.379244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.381962 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.381976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.383932 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.383938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.384132 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.384164 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:48:28.384243 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:48:28.384365 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7071,"mutex_wait_us":196,"spinlock_wait_cycles":3456,"thread_start_us":818,"threads_started":1}
I20260927 22:48:28.389278 11096 tablet_service.cc:1467] Processing CreateTablet for tablet 80c95be11a244b80893c0d8433fb3844 (DEFAULT_TABLE table=test-table [id=182dd0e9e337469b9867f5ea7ea4d2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:48:28.389227 10829 tablet_service.cc:1467] Processing CreateTablet for tablet 80c95be11a244b80893c0d8433fb3844 (DEFAULT_TABLE table=test-table [id=182dd0e9e337469b9867f5ea7ea4d2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:48:28.389191 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 80c95be11a244b80893c0d8433fb3844 (DEFAULT_TABLE table=test-table [id=182dd0e9e337469b9867f5ea7ea4d2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:48:28.390398 11096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c95be11a244b80893c0d8433fb3844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:48:28.390445 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c95be11a244b80893c0d8433fb3844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:48:28.390441 10829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c95be11a244b80893c0d8433fb3844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:48:28.401302 11179 tablet_bootstrap.cc:492] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Bootstrap starting.
I20260927 22:48:28.403815 11180 tablet_bootstrap.cc:492] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Bootstrap starting.
I20260927 22:48:28.404510 11181 tablet_bootstrap.cc:492] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Bootstrap starting.
I20260927 22:48:28.404701 11179 tablet_bootstrap.cc:654] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Neither blocks nor log segments found. Creating new log.
I20260927 22:48:28.405992 11179 log.cc:824] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Log is configured to *not* fsync() on all Append() calls
I20260927 22:48:28.407610 11180 tablet_bootstrap.cc:654] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Neither blocks nor log segments found. Creating new log.
I20260927 22:48:28.407851 11181 tablet_bootstrap.cc:654] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Neither blocks nor log segments found. Creating new log.
I20260927 22:48:28.408636 11180 log.cc:824] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:48:28.409046 11181 log.cc:824] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Log is configured to *not* fsync() on all Append() calls
I20260927 22:48:28.411306 11180 tablet_bootstrap.cc:492] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: No bootstrap required, opened a new log
I20260927 22:48:28.411572 11180 ts_tablet_manager.cc:1397] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260927 22:48:28.411661 11179 tablet_bootstrap.cc:492] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: No bootstrap required, opened a new log
I20260927 22:48:28.411739 11181 tablet_bootstrap.cc:492] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: No bootstrap required, opened a new log
I20260927 22:48:28.411937 11179 ts_tablet_manager.cc:1397] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 22:48:28.411979 11181 ts_tablet_manager.cc:1397] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:48:28.418627 11180 raft_consensus.cc:348] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.419008 11180 raft_consensus.cc:374] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:48:28.419130 11180 raft_consensus.cc:729] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7042458bda4f74bf28cab066a493a4, State: Initialized, Role: FOLLOWER
I20260927 22:48:28.419760 11180 consensus_queue.cc:260] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.420819 11028 heartbeater.cc:503] Master 127.0.97.62:34343 was elected leader, sending a full tablet report...
I20260927 22:48:28.421972 11179 raft_consensus.cc:348] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.421972 11181 raft_consensus.cc:348] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.422371 11181 raft_consensus.cc:374] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:48:28.422392 11179 raft_consensus.cc:374] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:48:28.422467 11179 raft_consensus.cc:729] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c985fbc25b34a10a76086b6cd7e3211, State: Initialized, Role: FOLLOWER
I20260927 22:48:28.422490 11181 raft_consensus.cc:729] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eada93b095b4ff588cf9dc47931cc21, State: Initialized, Role: FOLLOWER
I20260927 22:48:28.422585 11180 ts_tablet_manager.cc:1428] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260927 22:48:28.423027 11181 consensus_queue.cc:260] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.423079 11179 consensus_queue.cc:260] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.424738 11161 heartbeater.cc:503] Master 127.0.97.62:34343 was elected leader, sending a full tablet report...
I20260927 22:48:28.424988 11181 ts_tablet_manager.cc:1428] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260927 22:48:28.425318 11179 ts_tablet_manager.cc:1428] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260927 22:48:28.526661 11187 raft_consensus.cc:482] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:48:28.526969 11187 raft_consensus.cc:504] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.528664 11187 leader_election.cc:288] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eada93b095b4ff588cf9dc47931cc21 (127.0.97.1:36155), 2f7042458bda4f74bf28cab066a493a4 (127.0.97.2:42791)
I20260927 22:48:28.534577 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.529096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33913 (local address 127.0.97.1:36155)
0927 22:48:28.529303 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.529309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.529332 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:48:28.529409 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.529415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.529511 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.529621 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:48:28.530187 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.530926 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.532673 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.533760 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.533861 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.533935 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.534293 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.534390 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:48:28.534719 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.528867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36155 (local address 127.0.97.3:33913)
0927 22:48:28.529076 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:28.529096 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:28.529221 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:28.529802 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:28.529807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:28.529839 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:48:28.530026 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.530035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.531069 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.531075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.532507 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.532519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.533896 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.533903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.534101 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.534141 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:48:28.534251 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 22:48:28.534391 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":42}
I20260927 22:48:28.535424 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.529302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42791 (local address 127.0.97.3:34483)
0927 22:48:28.529938 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 22:48:28.529951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:48:28.530060 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:48:28.530344 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:48:28.530348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:48:28.530370 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:48:28.530648 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.530664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.531608 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.531611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.533201 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:48:28.533214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.534763 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.534769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:48:28.534932 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.534963 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:48:28.535092 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:48:28.535205 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":374,"threads_started":1}
I20260927 22:48:28.535552 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.529536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34483 (local address 127.0.97.2:42791)
0927 22:48:28.529732 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.529737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.530033 (+ 296us) server_negotiation.cc:407] Connection header received
0927 22:48:28.530187 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.530190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.530250 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.530319 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:48:28.530787 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.531476 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.533346 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.534512 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.534648 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.534754 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.535133 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.535268 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:48:28.535962 10849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c95be11a244b80893c0d8433fb3844" candidate_uuid: "5c985fbc25b34a10a76086b6cd7e3211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eada93b095b4ff588cf9dc47931cc21" is_pre_election: true
I20260927 22:48:28.536657 10849 raft_consensus.cc:2454] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c985fbc25b34a10a76086b6cd7e3211 in term 0.
I20260927 22:48:28.536659 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c95be11a244b80893c0d8433fb3844" candidate_uuid: "5c985fbc25b34a10a76086b6cd7e3211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7042458bda4f74bf28cab066a493a4" is_pre_election: true
I20260927 22:48:28.537195 10983 raft_consensus.cc:2454] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c985fbc25b34a10a76086b6cd7e3211 in term 0.
I20260927 22:48:28.537282 11051 leader_election.cc:302] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c985fbc25b34a10a76086b6cd7e3211, 5eada93b095b4ff588cf9dc47931cc21; no voters:
I20260927 22:48:28.537658 11187 raft_consensus.cc:2781] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:48:28.537788 11187 raft_consensus.cc:482] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:48:28.537896 11187 raft_consensus.cc:3037] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:48:28.540159 11187 raft_consensus.cc:504] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.540668 11187 leader_election.cc:288] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [CANDIDATE]: Term 1 election: Requested vote from peers 5eada93b095b4ff588cf9dc47931cc21 (127.0.97.1:36155), 2f7042458bda4f74bf28cab066a493a4 (127.0.97.2:42791)
I20260927 22:48:28.541121 10849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c95be11a244b80893c0d8433fb3844" candidate_uuid: "5c985fbc25b34a10a76086b6cd7e3211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eada93b095b4ff588cf9dc47931cc21"
I20260927 22:48:28.541237 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c95be11a244b80893c0d8433fb3844" candidate_uuid: "5c985fbc25b34a10a76086b6cd7e3211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7042458bda4f74bf28cab066a493a4"
I20260927 22:48:28.541296 10849 raft_consensus.cc:3037] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:48:28.541414 10983 raft_consensus.cc:3037] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:48:28.543376 10849 raft_consensus.cc:2454] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c985fbc25b34a10a76086b6cd7e3211 in term 1.
I20260927 22:48:28.543586 10983 raft_consensus.cc:2454] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c985fbc25b34a10a76086b6cd7e3211 in term 1.
I20260927 22:48:28.543797 11051 leader_election.cc:302] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c985fbc25b34a10a76086b6cd7e3211, 5eada93b095b4ff588cf9dc47931cc21; no voters:
I20260927 22:48:28.544052 11187 raft_consensus.cc:2781] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:48:28.544468 11187 raft_consensus.cc:686] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 1 LEADER]: Becoming Leader. State: Replica: 5c985fbc25b34a10a76086b6cd7e3211, State: Running, Role: LEADER
I20260927 22:48:28.544961 11187 consensus_queue.cc:237] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } }
I20260927 22:48:28.549196 10704 catalog_manager.cc:5495] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c985fbc25b34a10a76086b6cd7e3211 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5c985fbc25b34a10a76086b6cd7e3211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c985fbc25b34a10a76086b6cd7e3211" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 } health_report { overall_health: UNKNOWN } } }
W20260927 22:48:28.552753 11162 tablet.cc:2369] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:48:28.572281 11029 tablet.cc:2369] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:48:28.578102 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.572951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.0.97.3:44175)
0927 22:48:28.573121 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.573127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.573144 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:48:28.573357 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.573361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.573435 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.573568 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:48:28.573934 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.574634 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.576048 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.576572 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.576670 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.577111 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:48:28.577609 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:48:28.577713 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.577798 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.577861 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:48:28.585458 10983 raft_consensus.cc:1261] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 1 FOLLOWER]: Refusing update from remote peer 5c985fbc25b34a10a76086b6cd7e3211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:48:28.585510 10849 raft_consensus.cc:1261] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 1 FOLLOWER]: Refusing update from remote peer 5c985fbc25b34a10a76086b6cd7e3211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:48:28.586375 11189 consensus_queue.cc:1035] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eada93b095b4ff588cf9dc47931cc21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:48:28.586694 11187 consensus_queue.cc:1035] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f7042458bda4f74bf28cab066a493a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:48:28.599278 11194 mvcc.cc:205] Tried to move back new op lower bound from 7334089967957430272 to 7334089967803842560. Current Snapshot: MvccSnapshot[applied={T|T < 7334089967957430272}]
I20260927 22:48:28.611940 10703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:48:28.612248 10703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:48:28.615970 10703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fce0ed592cac47b58c29800049647344: Sending DeleteTablet for 3 replicas of tablet 80c95be11a244b80893c0d8433fb3844
I20260927 22:48:28.617002 11096 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c95be11a244b80893c0d8433fb3844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:48:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44272
I20260927 22:48:28.617105 10829 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c95be11a244b80893c0d8433fb3844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:48:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36936
I20260927 22:48:28.617174 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c95be11a244b80893c0d8433fb3844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:48:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59014
I20260927 22:48:28.619079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10766
I20260927 22:48:28.627882 11203 tablet_replica.cc:330] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: stopping tablet replica
I20260927 22:48:28.629042 11203 raft_consensus.cc:2229] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:48:28.630213 11203 raft_consensus.cc:2258] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:48:28.634413 11204 tablet_replica.cc:330] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: stopping tablet replica
I20260927 22:48:28.634944 11204 raft_consensus.cc:2229] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:48:28.635532 11204 raft_consensus.cc:2258] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:48:28.635967 11203 ts_tablet_manager.cc:1905] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:48:28.637512 11204 ts_tablet_manager.cc:1905] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:48:28.649024 11204 ts_tablet_manager.cc:1918] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:48:28.649709 11202 tablet_replica.cc:330] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: stopping tablet replica
I20260927 22:48:28.649916 11204 log.cc:1196] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/80c95be11a244b80893c0d8433fb3844
I20260927 22:48:28.650405 11203 ts_tablet_manager.cc:1918] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:48:28.650485 11204 ts_tablet_manager.cc:1939] T 80c95be11a244b80893c0d8433fb3844 P 5eada93b095b4ff588cf9dc47931cc21: Deleting consensus metadata
I20260927 22:48:28.650681 11203 log.cc:1196] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/80c95be11a244b80893c0d8433fb3844
I20260927 22:48:28.651266 11203 ts_tablet_manager.cc:1939] T 80c95be11a244b80893c0d8433fb3844 P 2f7042458bda4f74bf28cab066a493a4: Deleting consensus metadata
Sep 27 22:48:28 dist-test-slave-kr25 krb5kdc[10660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:48:28.652758 10691 catalog_manager.cc:4841] TS 5eada93b095b4ff588cf9dc47931cc21 (127.0.97.1:36155): tablet 80c95be11a244b80893c0d8433fb3844 (table test-table [id=182dd0e9e337469b9867f5ea7ea4d2e4]) successfully deleted
I20260927 22:48:28.653057 10691 catalog_manager.cc:4841] TS 2f7042458bda4f74bf28cab066a493a4 (127.0.97.2:42791): tablet 80c95be11a244b80893c0d8433fb3844 (table test-table [id=182dd0e9e337469b9867f5ea7ea4d2e4]) successfully deleted
I20260927 22:48:28.653249 11202 raft_consensus.cc:2229] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:48:28.653913 11202 raft_consensus.cc:2258] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:48:28.656085 11202 ts_tablet_manager.cc:1905] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:48:28.660336 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.622137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.0.97.1:36155)
0927 22:48:28.622317 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.622324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.640517 (+ 18193us) server_negotiation.cc:407] Connection header received
0927 22:48:28.640791 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.640796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.640877 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.640958 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:28.642724 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.643798 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.645595 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.646263 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.646360 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.653625 (+ 7265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:28.653638 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:28.653655 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:28.653693 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:28.657219 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.657990 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.657999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.658003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.658155 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.658565 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.658570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.658573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.658986 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:28.659276 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.659584 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.659785 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:48:28.661005 10889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:28.663012 11202 ts_tablet_manager.cc:1918] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:48:28.663331 11202 log.cc:1196] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/80c95be11a244b80893c0d8433fb3844
I20260927 22:48:28.664237 11202 ts_tablet_manager.cc:1939] T 80c95be11a244b80893c0d8433fb3844 P 5c985fbc25b34a10a76086b6cd7e3211: Deleting consensus metadata
I20260927 22:48:28.667150 10689 catalog_manager.cc:4841] TS 5c985fbc25b34a10a76086b6cd7e3211 (127.0.97.3:44175): tablet 80c95be11a244b80893c0d8433fb3844 (table test-table [id=182dd0e9e337469b9867f5ea7ea4d2e4]) successfully deleted
I20260927 22:48:28.754799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10766
I20260927 22:48:28.769266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10900
Sep 27 22:48:28 dist-test-slave-kr25 krb5kdc[10660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:48:28.784055 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.770036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.0.97.2:42791)
0927 22:48:28.770239 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.770245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.770278 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:48:28.770439 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.770445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.770532 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.770645 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:28.771860 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.772993 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.774188 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.774796 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.774918 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.778926 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:28.778939 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:28.778955 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:28.778986 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:28.781533 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.782123 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.782131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.782133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.782251 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.782584 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.782588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.782597 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.782936 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:28.783201 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.783526 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.783739 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:48:28.784675 11023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:28.872732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10900
I20260927 22:48:28.894697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 11033
Sep 27 22:48:28 dist-test-slave-kr25 krb5kdc[10660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:48:28.911928 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:48:28.895392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.0.97.3:44175)
0927 22:48:28.895576 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:48:28.895581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:48:28.895683 (+ 102us) server_negotiation.cc:407] Connection header received
0927 22:48:28.895919 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:48:28.895924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:48:28.896005 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:48:28.896116 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:48:28.897789 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.898666 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.900400 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:48:28.900852 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:48:28.900952 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:48:28.906213 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:48:28.906224 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:48:28.906239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:48:28.906277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:48:28.909115 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.909876 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.909883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.909885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.910074 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:48:28.910450 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:48:28.910454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:48:28.910456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:48:28.910845 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:48:28.911116 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 22:48:28.911388 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:48:28.911584 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:48:28.912618 11156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:29.029381 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 11033
I20260927 22:48:29.044623 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 10673
I20260927 22:48:29.045653 10735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:48:29.149310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 10673
2026-09-27T22:48:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3974 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:48:29 dist-test-slave-kr25 krb5kdc[11212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:48:29 dist-test-slave-kr25 krb5kdc[11212](info): set up 2 sockets
Sep 27 22:48:29 dist-test-slave-kr25 krb5kdc[11212](info): commencing operation
krb5kdc: starting...
W20260927 22:48:30.168087 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:48:30 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:48:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 22:48:30.450601 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfHsC0g/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/postgres -l logfile start
2026-09-27 22:48:32.859 UTC [11242] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:48:32.859 UTC [11242] LOG: listening on IPv4 address "127.0.97.20", port 45813
2026-09-27 22:48:32.863 UTC [11242] LOG: listening on Unix socket "/tmp/.s.PGSQL.45813"
2026-09-27 22:48:32.883 UTC [11245] LOG: database system was shut down at 2026-09-27 22:48:31 UTC
2026-09-27 22:48:32.886 UTC [11242] LOG: database system is ready to accept connections
I20260927 22:48:33.767750 388 mini_postgres.cc:96] Postgres bound to 45813
2026-09-27 22:48:33.783 UTC [11253] FATAL: database "slave" does not exist
127.0.97.20:45813 - accepting connections
I20260927 22:48:33.783978 388 mini_ranger.cc:161] Starting Ranger...
I20260927 22:48:33.807022 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:48:34.281958 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:48:34.282068 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-admin
2026-09-27 22:48:34,562 [I] DB FLAVOR :POSTGRES
2026-09-27 22:48:34,562 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:48:34,562 [I] Checking connection..
2026-09-27 22:48:34,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:48:34,819 [I] Checking connection passed.
2026-09-27 22:48:34,819 [I] --------- Verifying version history table ---------
2026-09-27 22:48:34,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:48:35,056 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:48:35,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:48:35,292 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:48:35,292 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:48:35,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:48:35.537 UTC [11317] WARNING: there is no transaction in progress
2026-09-27 22:48:35,543 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:48:35,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:48:35,779 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:48:35,780 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:48:35,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:48:36,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:48:36,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kr25.c.gcp-upstream.internal','N') ;"
2026-09-27 22:48:36,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:48:36,713 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:48:36,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:48:36,963 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:48:36,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:48:37,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:48:37,462 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:48:37,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:48:38.407 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.423 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.439 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.454 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.470 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.488 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.550 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.557 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.570 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.580 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.591 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.604 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.613 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.619 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.625 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:38.632 UTC [11425] WARNING: there is no transaction in progress
2026-09-27 22:48:39,243 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:48:39,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:48:39,465 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:48:39,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:48:39,697 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:48:39,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:48:39,944 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:48:39,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:48:40,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:48:40,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kr25.c.gcp-upstream.internal';"
2026-09-27 22:48:40,669 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:48:40,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:48:40,902 [I] DB_PATCHES have already been applied
I20260927 22:48:40.907768 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfHsC0g/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:48:40.907861 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 22:48:40.920657 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:53149
Sep 27, 2026 10:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:48:41.970407 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.186s user 0.006s sys 0.030s
Sep 27, 2026 10:49:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:49:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:49:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:49:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:49:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:49:03.428216 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:49:03 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549343, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:49:03.582048 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:49:03.601374 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:49:04.454809 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:49:04.454957 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms
I20260927 22:49:04.455066 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:45813
2026-09-27 22:49:04,857 [I] DB FLAVOR :POSTGRES
2026-09-27 22:49:04,857 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:49:04,857 [I] Checking connection
2026-09-27 22:49:04,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:49:05,086 [I] connection success
2026-09-27 22:49:05,086 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:49:05,086 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:49:05,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:49:05,327 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:49:05,327 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:49:05,328 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:49:05,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:49:05,568 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:49:05.989434 388 mini_ranger_kms.cc:310] Created kms service
I20260927 22:49:06.165319 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:49:06.207537 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:49:06 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549343, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:49:06.582252 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:49:06.582520 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:49:06.582566 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 22:49:06.603955 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11605
I20260927 22:49:06.604104 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:34401
Sep 27, 2026 10:49:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:49:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:49:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:49:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:49:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:49:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 22:49:08 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549348, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:49:08.675689 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.094s user 0.102s sys 0.322s
I20260927 22:49:08.675932 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:49:08.676142 388 mini_ranger_kms.cc:401] 127.0.97.20:34401/kms/v1/keys
Sep 27 22:49:09 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549348, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:49:09 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549348, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:49:11 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:49:11.680998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32997
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32997
--ranger_config_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34401/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:49:12.008705 11656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:49:12.008983 11656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:49:12.009091 11656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:49:12.016510 11656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:49:12.016610 11656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:49:12.016676 11656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:49:12.016743 11656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:49:12.016809 11656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:49:12.026595 11656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:34401/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32997
--ranger_config_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:49:12.027978 11656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:49:12.029305 11656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:49:12.041188 11661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:12.041143 11664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:12.041236 11662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:49:12.041282 11656 server_base.cc:1034] running on GCE node
I20260927 22:49:12.042495 11656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:49:12.043931 11656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:49:12.045133 11656 hybrid_clock.cc:648] HybridClock initialized: now 1790549352045088 us; error 28 us; skew 500 ppm
I20260927 22:49:12.045498 11656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:49:12 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:49:12.052210 11656 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:49:12.053810 11656 webserver.cc:460] Webserver started at http://127.0.97.62:41341/ using document root <none> and password file <none>
I20260927 22:49:12.054504 11656 fs_manager.cc:359] Metadata directory not provided
I20260927 22:49:12.054623 11656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:49:12.054940 11656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:49:12 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:49:12.113904 11656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "0a8aada5d71a4c13aedd38955dbe59a8"
format_stamp: "Formatted at 2026-09-27 22:49:12 on dist-test-slave-kr25"
server_key: "d007e333a2486ae933d06b7c3fd16b90"
server_key_iv: "e34e9dd177510a54bfa061d3d01608f7"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:12.114722 11656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "0a8aada5d71a4c13aedd38955dbe59a8"
format_stamp: "Formatted at 2026-09-27 22:49:12 on dist-test-slave-kr25"
server_key: "d007e333a2486ae933d06b7c3fd16b90"
server_key_iv: "e34e9dd177510a54bfa061d3d01608f7"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:12.119900 11656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:49:12.139222 11671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:12.141028 11656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:49:12.141227 11656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "0a8aada5d71a4c13aedd38955dbe59a8"
format_stamp: "Formatted at 2026-09-27 22:49:12 on dist-test-slave-kr25"
server_key: "d007e333a2486ae933d06b7c3fd16b90"
server_key_iv: "e34e9dd177510a54bfa061d3d01608f7"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:12.141507 11656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:49:12.161559 11656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:49:12.165457 11656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:49:12.165726 11656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:49:12.184561 11656 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32997
I20260927 22:49:12.184607 11722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32997 every 8 connection(s)
I20260927 22:49:12.185904 11656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:49:12.189965 11723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:49:12.193985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 11656
I20260927 22:49:12.194187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:49:12.194958 388 external_mini_cluster.cc:1428] Setting key fa2dc919886240c319fa415615fb41ba
I20260927 22:49:12.203516 11723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8: Bootstrap starting.
I20260927 22:49:12.207360 11723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8: Neither blocks nor log segments found. Creating new log.
I20260927 22:49:12.208379 11723 log.cc:824] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:49:12 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:49:12.212110 11723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8: No bootstrap required, opened a new log
I20260927 22:49:12.217588 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:12.197008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.0.97.62:32997)
0927 22:49:12.197750 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:12.197762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:12.197802 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:49:12.199336 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:12.199358 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:12.199801 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:12.200128 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:12.201778 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:12.203028 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:12.204790 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:12.205641 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:12.205800 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:12.209931 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:12.209939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:12.209962 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:12.209995 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:12.213520 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:12.214095 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:12.214103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:12.214111 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:12.214298 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:12.214608 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:12.214613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:12.214616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:12.215261 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:12.215586 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:12.215875 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:49:12.216256 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":360,"threads_started":1}
I20260927 22:49:12.221386 11723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260927 22:49:12.221694 11723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:49:12.221827 11723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8aada5d71a4c13aedd38955dbe59a8, State: Initialized, Role: FOLLOWER
I20260927 22:49:12.222358 11723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260927 22:49:12.222574 11723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:49:12.222690 11723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:49:12.222846 11723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:49:12.225286 11723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260927 22:49:12.225747 11723 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a8aada5d71a4c13aedd38955dbe59a8; no voters:
I20260927 22:49:12.226329 11723 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:49:12.226456 11728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:49:12.227089 11728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [term 1 LEADER]: Becoming Leader. State: Replica: 0a8aada5d71a4c13aedd38955dbe59a8, State: Running, Role: LEADER
I20260927 22:49:12.227655 11728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } }
I20260927 22:49:12.228307 11723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:49:12.230227 11729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } } }
I20260927 22:49:12.230238 11730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8aada5d71a4c13aedd38955dbe59a8. Latest consensus state: current_term: 1 leader_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8aada5d71a4c13aedd38955dbe59a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32997 } } }
I20260927 22:49:12.230647 11729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:49:12.230692 11730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:49:12.238276 11734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:49:12.243574 11734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:49:12.244235 11723 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:49:12.260830 11734 catalog_manager.cc:1347] Generated new cluster ID: 0984626857e249ed858762e155643f97
I20260927 22:49:12.261055 11734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:49:12.276026 11734 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:49:12.278009 11734 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:49:12.286965 11734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8: Generated new TSK 0
I20260927 22:49:12.288100 11734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:49:13.035334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34401/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32997
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:49:13 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:49:13.465571 11768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:49:13.466013 11768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:49:13.466177 11768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:49:13.477501 11768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:49:13.477645 11768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:49:13.477841 11768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:49:13.493630 11768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:34401/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:49:13.495373 11768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:49:13.497148 11768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:49:13.514642 11778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:13.516149 11779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:49:13.524220 11768 server_base.cc:1034] running on GCE node
W20260927 22:49:13.524948 11781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:49:13.525801 11768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:49:13.527108 11768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:49:13.528402 11768 hybrid_clock.cc:648] HybridClock initialized: now 1790549353528331 us; error 56 us; skew 500 ppm
I20260927 22:49:13.528800 11768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:49:13 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:49:13.533593 11768 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:49:13.535763 11768 webserver.cc:460] Webserver started at http://127.0.97.1:45619/ using document root <none> and password file <none>
I20260927 22:49:13.536720 11768 fs_manager.cc:359] Metadata directory not provided
I20260927 22:49:13.536872 11768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:49:13.537248 11768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:49:13 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:49:13.558099 11768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "bb4bc7e584d140c0ae557b76a8abb954"
format_stamp: "Formatted at 2026-09-27 22:49:13 on dist-test-slave-kr25"
server_key: "940c5889bdfddcdaa35ec4742ea5086f"
server_key_iv: "528cb5db2f5213c8cba85928cd7e8eee"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:13.559283 11768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bb4bc7e584d140c0ae557b76a8abb954"
format_stamp: "Formatted at 2026-09-27 22:49:13 on dist-test-slave-kr25"
server_key: "940c5889bdfddcdaa35ec4742ea5086f"
server_key_iv: "528cb5db2f5213c8cba85928cd7e8eee"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:13.573890 11768 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260927 22:49:13.599978 11788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:13.601487 11768 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260927 22:49:13.601660 11768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "bb4bc7e584d140c0ae557b76a8abb954"
format_stamp: "Formatted at 2026-09-27 22:49:13 on dist-test-slave-kr25"
server_key: "940c5889bdfddcdaa35ec4742ea5086f"
server_key_iv: "528cb5db2f5213c8cba85928cd7e8eee"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:13.601879 11768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:49:13.628984 11768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:49:13.634403 11768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:49:13.634801 11768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:49:13.635885 11768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:49:13.637936 11768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:49:13.638031 11768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:13.638150 11768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:49:13.638206 11768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:13.678845 11768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41627
I20260927 22:49:13.678879 11900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41627 every 8 connection(s)
I20260927 22:49:13.680591 11768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:49:13.683249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 11768
I20260927 22:49:13.683470 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:49:13.684240 388 external_mini_cluster.cc:1428] Setting key be2672a397d7f6f08974ee5e048f2245
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 27 22:49:13 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:49:13.729434 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:13.683938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50793 (local address 127.0.97.62:32997)
0927 22:49:13.703845 (+ 19907us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:13.703852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:13.707223 (+ 3371us) server_negotiation.cc:407] Connection header received
0927 22:49:13.707284 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:13.707288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:13.707373 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:13.707504 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:13.709701 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:13.711393 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:13.713772 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:13.714411 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:13.714555 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:13.721236 (+ 6681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:13.721243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:13.721248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:13.721287 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:13.724793 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:13.725732 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:13.725737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:13.725741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:13.725894 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:13.726427 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:13.726432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:13.726435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:13.726789 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:13.726923 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:13.728781 (+ 1858us) server_negotiation.cc:299] Negotiation successful
0927 22:49:13.729013 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19749,"thread_start_us":19620,"threads_started":1}
I20260927 22:49:13.731103 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:13.684506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32997 (local address 127.0.97.1:50793)
0927 22:49:13.704433 (+ 19927us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:13.704486 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:13.706832 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:13.707801 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:13.707814 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:13.708648 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:49:13.709480 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:13.709507 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:13.711616 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:13.711622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:13.713600 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:13.713615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:13.714844 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:13.714852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:13.715177 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:13.715968 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:49:13.715998 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:49:13.720967 (+ 4969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:49:13.724984 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:49:13.724993 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:49:13.725011 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:49:13.725590 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:49:13.726029 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:49:13.726034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:49:13.726038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:49:13.726271 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:49:13.727005 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:49:13.727013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:49:13.727471 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 22:49:13.727815 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 22:49:13.728398 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19628,"thread_start_us":19411,"threads_started":1}
I20260927 22:49:13.733202 11901 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32997
I20260927 22:49:13.733515 11901 heartbeater.cc:464] Registering TS with master...
I20260927 22:49:13.734335 11901 heartbeater.cc:511] Master 127.0.97.62:32997 requested a full tablet report, sending...
I20260927 22:49:13.736866 11688 ts_manager.cc:194] Registered new tserver with Master: bb4bc7e584d140c0ae557b76a8abb954 (127.0.97.1:41627)
I20260927 22:49:13.738822 11688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:49:13.756595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34401/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32997
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:49:13 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:49:14 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 22:49:14.179384 11907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:49:14.179802 11907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:49:14.179973 11907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:49:14.189531 11907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:49:14.189769 11907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:49:14.190026 11907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:49:14.205756 11907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:34401/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:49:14.207715 11907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:49:14.209594 11907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:49:14.243898 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:14.245496 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:14.251645 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:49:14.252578 11907 server_base.cc:1034] running on GCE node
I20260927 22:49:14.253202 11907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:49:14.254210 11907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:49:14.257550 11907 hybrid_clock.cc:648] HybridClock initialized: now 1790549354257430 us; error 125 us; skew 500 ppm
I20260927 22:49:14.257979 11907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:49:14 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:49:14.262182 11907 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:49:14.271628 11907 webserver.cc:460] Webserver started at http://127.0.97.2:36459/ using document root <none> and password file <none>
I20260927 22:49:14.272601 11907 fs_manager.cc:359] Metadata directory not provided
I20260927 22:49:14.272773 11907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:49:14.273144 11907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:49:14 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:49:14.291828 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "0c21761d86714715b9bf0c22fae8bfff"
format_stamp: "Formatted at 2026-09-27 22:49:14 on dist-test-slave-kr25"
server_key: "b03768facac19362fcbfd1dcad078ba2"
server_key_iv: "d03e8a5a0520de9a2c6890d1e67c241f"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:14.292972 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0c21761d86714715b9bf0c22fae8bfff"
format_stamp: "Formatted at 2026-09-27 22:49:14 on dist-test-slave-kr25"
server_key: "b03768facac19362fcbfd1dcad078ba2"
server_key_iv: "d03e8a5a0520de9a2c6890d1e67c241f"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:14.323127 11907 fs_manager.cc:691] Time spent creating directory manager: real 0.030s user 0.013s sys 0.015s
I20260927 22:49:14.365775 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:14.367874 11907 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260927 22:49:14.368073 11907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "0c21761d86714715b9bf0c22fae8bfff"
format_stamp: "Formatted at 2026-09-27 22:49:14 on dist-test-slave-kr25"
server_key: "b03768facac19362fcbfd1dcad078ba2"
server_key_iv: "d03e8a5a0520de9a2c6890d1e67c241f"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:14.368371 11907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:49:14.453598 11907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:49:14.459712 11907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:49:14.460069 11907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:49:14.461275 11907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:49:14.463243 11907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:49:14.463344 11907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:14.463542 11907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:49:14.463637 11907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:14.504395 11907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34957
I20260927 22:49:14.504437 12043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34957 every 8 connection(s)
I20260927 22:49:14.506068 11907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:49:14.510582 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 11907
I20260927 22:49:14.510757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:49:14.511313 388 external_mini_cluster.cc:1428] Setting key 9a1d42d0e0ebb948d695fbf6872da188
Sep 27 22:49:14 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:49:14.544441 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:14.509477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48077 (local address 127.0.97.62:32997)
0927 22:49:14.520609 (+ 11132us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:14.520615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:14.524162 (+ 3547us) server_negotiation.cc:407] Connection header received
0927 22:49:14.526330 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:14.526336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:14.526438 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:14.526537 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:14.528745 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:14.529950 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:14.532377 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:14.533038 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:14.533134 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:14.537636 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:14.537644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:14.537649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:14.537686 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:14.540962 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:14.541879 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:14.541884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:14.541887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:14.542044 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:14.542493 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:14.542498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:14.542501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:14.542851 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:14.542986 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:14.543845 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 22:49:14.544071 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11012,"thread_start_us":10910,"threads_started":1}
I20260927 22:49:14.547020 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:14.509629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32997 (local address 127.0.97.2:48077)
0927 22:49:14.521155 (+ 11526us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:14.521210 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:14.525865 (+ 4655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:14.527004 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:14.527016 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:14.527653 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:49:14.528553 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:14.528578 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:14.530160 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:14.530165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:14.532207 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:14.532222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:14.533464 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:14.533470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:14.533852 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:14.534678 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:49:14.534708 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:49:14.537393 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:49:14.541089 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:49:14.541097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:49:14.541115 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:49:14.541746 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:49:14.542146 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:49:14.542151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:49:14.542154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:49:14.542368 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:49:14.542978 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:49:14.542986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:49:14.543384 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 22:49:14.544943 (+ 1559us) client_negotiation.cc:240] Negotiation successful
0927 22:49:14.545461 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11208,"thread_start_us":13719,"threads_started":1}
I20260927 22:49:14.548945 12044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32997
I20260927 22:49:14.549245 12044 heartbeater.cc:464] Registering TS with master...
I20260927 22:49:14.550050 12044 heartbeater.cc:511] Master 127.0.97.62:32997 requested a full tablet report, sending...
I20260927 22:49:14.552186 11688 ts_manager.cc:194] Registered new tserver with Master: 0c21761d86714715b9bf0c22fae8bfff (127.0.97.2:34957)
I20260927 22:49:14.553225 11688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:49:14.576193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34401/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32997
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:49:14.741890 11901 heartbeater.cc:503] Master 127.0.97.62:32997 was elected leader, sending a full tablet report...
W20260927 22:49:14.978868 12050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:49:14.979125 12050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:49:14.979238 12050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:49:14.986166 12050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:49:14.986274 12050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:49:14.986423 12050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:49:14.996487 12050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:34401/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:49:14.997712 12050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:49:14.998941 12050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:49:15.011710 12056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:15.011785 12055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:15.012388 12058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:49:15.012707 12050 server_base.cc:1034] running on GCE node
I20260927 22:49:15.013262 12050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:49:15.014189 12050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:49:15.015401 12050 hybrid_clock.cc:648] HybridClock initialized: now 1790549355015339 us; error 48 us; skew 500 ppm
I20260927 22:49:15.015712 12050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:49:15 dist-test-slave-kr25 krb5kdc[11212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:49:15.018575 12050 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:49:15.019984 12050 webserver.cc:460] Webserver started at http://127.0.97.3:44739/ using document root <none> and password file <none>
I20260927 22:49:15.020594 12050 fs_manager.cc:359] Metadata directory not provided
I20260927 22:49:15.020687 12050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:49:15.020943 12050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:49:15 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:49:15.037374 12050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "a577079d3dc74242a7269e083ee62616"
format_stamp: "Formatted at 2026-09-27 22:49:15 on dist-test-slave-kr25"
server_key: "caf3d3fb4e120fe724996a11f245af3e"
server_key_iv: "6a6f883f2f3de2f8abd03e93bd013d47"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:15.038187 12050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a577079d3dc74242a7269e083ee62616"
format_stamp: "Formatted at 2026-09-27 22:49:15 on dist-test-slave-kr25"
server_key: "caf3d3fb4e120fe724996a11f245af3e"
server_key_iv: "6a6f883f2f3de2f8abd03e93bd013d47"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:15.043398 12050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:49:15.060981 12065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:15.062669 12050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:49:15.062842 12050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "a577079d3dc74242a7269e083ee62616"
format_stamp: "Formatted at 2026-09-27 22:49:15 on dist-test-slave-kr25"
server_key: "caf3d3fb4e120fe724996a11f245af3e"
server_key_iv: "6a6f883f2f3de2f8abd03e93bd013d47"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:15.063098 12050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:49:15.076884 12050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:49:15.080315 12050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:49:15.080559 12050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:49:15.081358 12050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:49:15.082609 12050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:49:15.082677 12050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:15.082760 12050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:49:15.082800 12050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:15.114238 12050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40573
I20260927 22:49:15.114274 12177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40573 every 8 connection(s)
I20260927 22:49:15.115499 12050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:49:15.121178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 12050
I20260927 22:49:15.121389 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:49:15.121908 388 external_mini_cluster.cc:1428] Setting key e0d9f9d1643825cd0eb3403bd86f8514
Sep 27 22:49:15 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:49:15.134259 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.118200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40269 (local address 127.0.97.62:32997)
0927 22:49:15.118617 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.118621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.119510 (+ 889us) server_negotiation.cc:407] Connection header received
0927 22:49:15.121380 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.121384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.121470 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.121564 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:15.123245 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.123971 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.125433 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.125871 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.125974 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.129090 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:15.129095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:15.129098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:15.129128 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:15.131669 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.132287 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.132291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.132293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.132407 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.132775 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.132779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.132781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.133054 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:15.133177 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.133850 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.134003 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":193,"threads_started":1}
I20260927 22:49:15.135591 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.118734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32997 (local address 127.0.97.3:40269)
0927 22:49:15.119354 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.119391 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:15.121052 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:15.121802 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:15.121813 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:15.122376 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:49:15.123047 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.123069 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.124114 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.124118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.125292 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.125304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.126062 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.126069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.126283 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.126890 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:49:15.126913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:49:15.128903 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:49:15.131782 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:49:15.131788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:49:15.131800 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:49:15.132199 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:49:15.132479 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:49:15.132482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:49:15.132484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:49:15.132665 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:49:15.133170 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:49:15.133176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:49:15.133516 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 22:49:15.133735 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:49:15.134096 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":239,"threads_started":1}
I20260927 22:49:15.136818 12178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32997
I20260927 22:49:15.137095 12178 heartbeater.cc:464] Registering TS with master...
I20260927 22:49:15.137754 12178 heartbeater.cc:511] Master 127.0.97.62:32997 requested a full tablet report, sending...
I20260927 22:49:15.139086 11687 ts_manager.cc:194] Registered new tserver with Master: a577079d3dc74242a7269e083ee62616 (127.0.97.3:40573)
I20260927 22:49:15.139952 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40269
I20260927 22:49:15.149287 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:49:15.166047 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.153788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.0.97.62:32997)
0927 22:49:15.153969 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.153974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.154418 (+ 444us) server_negotiation.cc:407] Connection header received
0927 22:49:15.154535 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.154539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.154627 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.154711 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:15.156075 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.156779 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.158349 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.158774 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.158865 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.161175 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:15.161180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:15.161183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:15.161218 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:15.164023 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.164595 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.164599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.164601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.164722 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.164990 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.164993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.164995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.165226 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:15.165390 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.165664 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.165814 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:49:15.170025 11687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:49:15.177148 11687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:49:15.224220 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.190724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.0.97.1:41627)
0927 22:49:15.208060 (+ 17336us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.208072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.216166 (+ 8094us) server_negotiation.cc:407] Connection header received
0927 22:49:15.216626 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.216635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.216888 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.217060 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:49:15.217980 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.219515 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.221988 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.222942 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.223067 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.223295 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.223713 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.223898 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17105,"thread_start_us":17352,"threads_started":1}
I20260927 22:49:15.224291 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.190248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57644)
0927 22:49:15.216042 (+ 25794us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.216060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:15.216422 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:15.217370 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:15.217377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:15.217406 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:49:15.217799 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.217809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.219737 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.219741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.221855 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.221865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.223160 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.223167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.223379 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.223517 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 22:49:15.223659 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:49:15.223853 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25674,"mutex_wait_us":61,"thread_start_us":18205,"threads_started":1}
I20260927 22:49:15.227653 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.209524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34957 (local address 127.0.0.1:57184)
0927 22:49:15.218987 (+ 9463us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.219004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:15.219102 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:15.221011 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:15.221018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:15.221042 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:49:15.221440 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.221451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.223221 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.223225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.225163 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.225176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.226713 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.226719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.226878 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.226921 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:49:15.227012 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:49:15.227134 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9335,"thread_start_us":6656,"threads_started":1}
I20260927 22:49:15.227677 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.209634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.0.97.2:34957)
0927 22:49:15.220476 (+ 10842us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.220482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.220505 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:49:15.220573 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.220578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.220754 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.220952 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:49:15.221621 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.223071 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.225302 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.226535 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.226668 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.226830 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.227104 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.227316 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10707,"thread_start_us":10592,"threads_started":1}
I20260927 22:49:15.228614 11836 tablet_service.cc:1467] Processing CreateTablet for tablet 13e56fe0f84f433cbcd6d35486d4c34a (DEFAULT_TABLE table=test-table [id=b64040481e36404a92666911a42a9994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:49:15.229609 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.189657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40573 (local address 127.0.0.1:60594)
0927 22:49:15.208868 (+ 19211us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.208905 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:15.209096 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:15.220738 (+ 11642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:15.220743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:15.220765 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:49:15.221227 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.221240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.222992 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.222997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.225236 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.225251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.228810 (+ 3559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.228817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.229025 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.229056 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:49:15.229153 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:49:15.229282 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18958,"thread_start_us":29061,"threads_started":1}
I20260927 22:49:15.229664 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.189732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.97.3:40573)
0927 22:49:15.220134 (+ 30402us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.220142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.220184 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:49:15.220270 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.220278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.220537 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.220733 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:49:15.221395 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.222856 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.225418 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.228613 (+ 3195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.228729 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.228965 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.229202 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.229350 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30232,"thread_start_us":30094,"threads_started":1}
I20260927 22:49:15.230019 11836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e56fe0f84f433cbcd6d35486d4c34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:49:15.234970 11979 tablet_service.cc:1467] Processing CreateTablet for tablet 13e56fe0f84f433cbcd6d35486d4c34a (DEFAULT_TABLE table=test-table [id=b64040481e36404a92666911a42a9994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:49:15.236481 11979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e56fe0f84f433cbcd6d35486d4c34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:49:15.235917 12113 tablet_service.cc:1467] Processing CreateTablet for tablet 13e56fe0f84f433cbcd6d35486d4c34a (DEFAULT_TABLE table=test-table [id=b64040481e36404a92666911a42a9994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:49:15.237393 12113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e56fe0f84f433cbcd6d35486d4c34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:49:15.241353 12197 tablet_bootstrap.cc:492] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: Bootstrap starting.
I20260927 22:49:15.244372 12197 tablet_bootstrap.cc:654] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: Neither blocks nor log segments found. Creating new log.
I20260927 22:49:15.245301 12197 log.cc:824] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: Log is configured to *not* fsync() on all Append() calls
I20260927 22:49:15.247839 12197 tablet_bootstrap.cc:492] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: No bootstrap required, opened a new log
I20260927 22:49:15.248091 12197 ts_tablet_manager.cc:1397] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 22:49:15.251718 12199 tablet_bootstrap.cc:492] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Bootstrap starting.
I20260927 22:49:15.253099 12200 tablet_bootstrap.cc:492] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Bootstrap starting.
I20260927 22:49:15.255204 12199 tablet_bootstrap.cc:654] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Neither blocks nor log segments found. Creating new log.
I20260927 22:49:15.256209 12200 tablet_bootstrap.cc:654] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Neither blocks nor log segments found. Creating new log.
I20260927 22:49:15.256390 12199 log.cc:824] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Log is configured to *not* fsync() on all Append() calls
I20260927 22:49:15.257236 12200 log.cc:824] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Log is configured to *not* fsync() on all Append() calls
I20260927 22:49:15.257588 12197 raft_consensus.cc:348] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.258002 12197 raft_consensus.cc:374] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:49:15.258112 12197 raft_consensus.cc:729] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4bc7e584d140c0ae557b76a8abb954, State: Initialized, Role: FOLLOWER
I20260927 22:49:15.258724 12197 consensus_queue.cc:260] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.259475 12199 tablet_bootstrap.cc:492] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: No bootstrap required, opened a new log
I20260927 22:49:15.259918 12200 tablet_bootstrap.cc:492] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: No bootstrap required, opened a new log
I20260927 22:49:15.260362 12197 ts_tablet_manager.cc:1428] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 22:49:15.260576 12200 ts_tablet_manager.cc:1397] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260927 22:49:15.260725 12199 ts_tablet_manager.cc:1397] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:49:15.268486 12199 raft_consensus.cc:348] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.268905 12199 raft_consensus.cc:374] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:49:15.269003 12199 raft_consensus.cc:729] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c21761d86714715b9bf0c22fae8bfff, State: Initialized, Role: FOLLOWER
I20260927 22:49:15.269112 12200 raft_consensus.cc:348] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.269541 12200 raft_consensus.cc:374] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:49:15.269636 12200 raft_consensus.cc:729] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a577079d3dc74242a7269e083ee62616, State: Initialized, Role: FOLLOWER
I20260927 22:49:15.269529 12199 consensus_queue.cc:260] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.270637 12200 consensus_queue.cc:260] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.270955 12044 heartbeater.cc:503] Master 127.0.97.62:32997 was elected leader, sending a full tablet report...
I20260927 22:49:15.271586 12199 ts_tablet_manager.cc:1428] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 22:49:15.272390 12178 heartbeater.cc:503] Master 127.0.97.62:32997 was elected leader, sending a full tablet report...
I20260927 22:49:15.272994 12200 ts_tablet_manager.cc:1428] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
W20260927 22:49:15.367457 12179 tablet.cc:2369] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:49:15.414621 12203 raft_consensus.cc:482] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:49:15.414889 12203 raft_consensus.cc:504] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.416538 12203 leader_election.cc:288] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a577079d3dc74242a7269e083ee62616 (127.0.97.3:40573), 0c21761d86714715b9bf0c22fae8bfff (127.0.97.2:34957)
I20260927 22:49:15.423655 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.416881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43583 (local address 127.0.97.3:40573)
0927 22:49:15.417087 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.417094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.417801 (+ 707us) server_negotiation.cc:407] Connection header received
0927 22:49:15.417974 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.417980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.418053 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.418120 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:49:15.418756 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.419733 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.421673 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.422720 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.422823 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.422902 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.423276 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.423399 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:49:15.423775 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.416929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40573 (local address 127.0.97.1:43583)
0927 22:49:15.417686 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.417705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:15.417836 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:15.418158 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:15.418162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:15.418185 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:49:15.418592 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.418605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.419846 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.419851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.421526 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.421541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.422926 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.422932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.423100 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.423157 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:49:15.423228 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:49:15.423314 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":66,"spinlock_wait_cycles":1792,"thread_start_us":398,"threads_started":1}
I20260927 22:49:15.423781 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.417056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38371 (local address 127.0.97.2:34957)
0927 22:49:15.417261 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.417268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.417883 (+ 615us) server_negotiation.cc:407] Connection header received
0927 22:49:15.418055 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.418059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.418122 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.418206 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:49:15.419314 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.420168 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.421978 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.422885 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.423023 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.423110 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.423405 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.423556 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:49:15.423880 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.417052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34957 (local address 127.0.97.1:38371)
0927 22:49:15.417770 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.417788 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:49:15.417926 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:49:15.418244 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:49:15.418248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:49:15.418279 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:49:15.419163 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.419178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.420301 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.420305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.421835 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:49:15.421849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.423113 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.423118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:49:15.423257 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.423288 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:49:15.423380 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:49:15.423516 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":44,"thread_start_us":435,"threads_started":1}
I20260927 22:49:15.424868 12133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e56fe0f84f433cbcd6d35486d4c34a" candidate_uuid: "bb4bc7e584d140c0ae557b76a8abb954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a577079d3dc74242a7269e083ee62616" is_pre_election: true
I20260927 22:49:15.424934 11999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e56fe0f84f433cbcd6d35486d4c34a" candidate_uuid: "bb4bc7e584d140c0ae557b76a8abb954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c21761d86714715b9bf0c22fae8bfff" is_pre_election: true
I20260927 22:49:15.425419 12133 raft_consensus.cc:2454] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4bc7e584d140c0ae557b76a8abb954 in term 0.
I20260927 22:49:15.425419 11999 raft_consensus.cc:2454] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4bc7e584d140c0ae557b76a8abb954 in term 0.
I20260927 22:49:15.426067 11792 leader_election.cc:302] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a577079d3dc74242a7269e083ee62616, bb4bc7e584d140c0ae557b76a8abb954; no voters:
I20260927 22:49:15.426440 12203 raft_consensus.cc:2781] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:49:15.426581 12203 raft_consensus.cc:482] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:49:15.426674 12203 raft_consensus.cc:3037] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:49:15.428952 12203 raft_consensus.cc:504] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.429535 12203 leader_election.cc:288] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [CANDIDATE]: Term 1 election: Requested vote from peers a577079d3dc74242a7269e083ee62616 (127.0.97.3:40573), 0c21761d86714715b9bf0c22fae8bfff (127.0.97.2:34957)
I20260927 22:49:15.430008 12133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e56fe0f84f433cbcd6d35486d4c34a" candidate_uuid: "bb4bc7e584d140c0ae557b76a8abb954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a577079d3dc74242a7269e083ee62616"
I20260927 22:49:15.430011 11999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e56fe0f84f433cbcd6d35486d4c34a" candidate_uuid: "bb4bc7e584d140c0ae557b76a8abb954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c21761d86714715b9bf0c22fae8bfff"
I20260927 22:49:15.430177 11999 raft_consensus.cc:3037] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:49:15.430199 12133 raft_consensus.cc:3037] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:49:15.432699 12133 raft_consensus.cc:2454] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4bc7e584d140c0ae557b76a8abb954 in term 1.
I20260927 22:49:15.432706 11999 raft_consensus.cc:2454] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4bc7e584d140c0ae557b76a8abb954 in term 1.
I20260927 22:49:15.433109 11791 leader_election.cc:302] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c21761d86714715b9bf0c22fae8bfff, bb4bc7e584d140c0ae557b76a8abb954; no voters:
I20260927 22:49:15.433394 12203 raft_consensus.cc:2781] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:49:15.433806 12203 raft_consensus.cc:686] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 1 LEADER]: Becoming Leader. State: Replica: bb4bc7e584d140c0ae557b76a8abb954, State: Running, Role: LEADER
I20260927 22:49:15.434334 12203 consensus_queue.cc:237] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } }
I20260927 22:49:15.438748 11688 catalog_manager.cc:5495] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb4bc7e584d140c0ae557b76a8abb954 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bb4bc7e584d140c0ae557b76a8abb954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb4bc7e584d140c0ae557b76a8abb954" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 } health_report { overall_health: UNKNOWN } } }
W20260927 22:49:15.459685 11902 tablet.cc:2369] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:49:15.490933 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.486064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.0.97.1:41627)
0927 22:49:15.486228 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.486232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.486249 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:49:15.486336 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.486339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.486440 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.486577 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:49:15.486856 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.487603 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.488958 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.489393 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.489501 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.489940 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:49:15.490458 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:49:15.490582 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.490644 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.490712 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:49:15.498992 12133 raft_consensus.cc:1261] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 1 FOLLOWER]: Refusing update from remote peer bb4bc7e584d140c0ae557b76a8abb954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:49:15.499006 11999 raft_consensus.cc:1261] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 1 FOLLOWER]: Refusing update from remote peer bb4bc7e584d140c0ae557b76a8abb954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:49:15.499874 12208 consensus_queue.cc:1035] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a577079d3dc74242a7269e083ee62616" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:49:15.500202 12203 consensus_queue.cc:1035] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c21761d86714715b9bf0c22fae8bfff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:49:15.509625 12045 tablet.cc:2369] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:49:15.537425 11687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:49:15.537658 11687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:49:15.540848 11687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a8aada5d71a4c13aedd38955dbe59a8: Sending DeleteTablet for 3 replicas of tablet 13e56fe0f84f433cbcd6d35486d4c34a
I20260927 22:49:15.541957 12113 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e56fe0f84f433cbcd6d35486d4c34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:49:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60594
I20260927 22:49:15.542279 11979 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e56fe0f84f433cbcd6d35486d4c34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:49:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57184
I20260927 22:49:15.542306 11836 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e56fe0f84f433cbcd6d35486d4c34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:49:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57644
I20260927 22:49:15.544500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 11768
I20260927 22:49:15.560752 12222 tablet_replica.cc:330] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: stopping tablet replica
I20260927 22:49:15.561770 12223 tablet_replica.cc:330] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: stopping tablet replica
I20260927 22:49:15.562211 12221 tablet_replica.cc:330] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: stopping tablet replica
I20260927 22:49:15.562245 12223 raft_consensus.cc:2229] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:49:15.562714 12221 raft_consensus.cc:2229] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:49:15.562813 12223 raft_consensus.cc:2258] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:49:15.563270 12222 raft_consensus.cc:2229] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:49:15.563794 12221 raft_consensus.cc:2258] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:49:15.564414 12222 raft_consensus.cc:2258] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:49:15.564802 12223 ts_tablet_manager.cc:1905] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:49:15.566530 12221 ts_tablet_manager.cc:1905] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:49:15 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:49:15.568420 12222 ts_tablet_manager.cc:1905] T 13e56fe0f84f433cbcd6d35486d4c34a P bb4bc7e584d140c0ae557b76a8abb954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:49:15.571341 12223 ts_tablet_manager.cc:1918] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:49:15.571537 12223 log.cc:1196] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/13e56fe0f84f433cbcd6d35486d4c34a
I20260927 22:49:15.572011 12223 ts_tablet_manager.cc:1939] T 13e56fe0f84f433cbcd6d35486d4c34a P 0c21761d86714715b9bf0c22fae8bfff: Deleting consensus metadata
I20260927 22:49:15.573751 11674 catalog_manager.cc:4841] TS 0c21761d86714715b9bf0c22fae8bfff (127.0.97.2:34957): tablet 13e56fe0f84f433cbcd6d35486d4c34a (table test-table [id=b64040481e36404a92666911a42a9994]) successfully deleted
I20260927 22:49:15.575739 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.545588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.0.97.1:41627)
0927 22:49:15.545753 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.545759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.552062 (+ 6303us) server_negotiation.cc:407] Connection header received
0927 22:49:15.552995 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.553002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.553091 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.553168 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:15.556884 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.558270 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.561310 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.561940 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.562055 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.567899 (+ 5844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:15.567913 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:15.567931 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:15.567978 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:15.571973 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.572886 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.572895 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.572899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.573112 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.573593 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.573601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.573605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.574106 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:15.574575 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.575139 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.575367 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:49:15.576433 11896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:49:15.580339 12221 ts_tablet_manager.cc:1918] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:49:15.580466 12221 log.cc:1196] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/13e56fe0f84f433cbcd6d35486d4c34a
I20260927 22:49:15.580789 12221 ts_tablet_manager.cc:1939] T 13e56fe0f84f433cbcd6d35486d4c34a P a577079d3dc74242a7269e083ee62616: Deleting consensus metadata
I20260927 22:49:15.582098 11675 catalog_manager.cc:4841] TS a577079d3dc74242a7269e083ee62616 (127.0.97.3:40573): tablet 13e56fe0f84f433cbcd6d35486d4c34a (table test-table [id=b64040481e36404a92666911a42a9994]) successfully deleted
I20260927 22:49:15.671372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 11768
W20260927 22:49:15.686110 11674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41627 (error 108)
I20260927 22:49:15.686343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 11907
I20260927 22:49:15.687772 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.687239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57668)
0927 22:49:15.687382 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.687602 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 22:49:15.688014 11674 catalog_manager.cc:4584] TS bb4bc7e584d140c0ae557b76a8abb954 (127.0.97.1:41627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13e56fe0f84f433cbcd6d35486d4c34a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Sep 27 22:49:15 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:49:15.700392 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.687066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.0.97.2:34957)
0927 22:49:15.687216 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.687220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.687332 (+ 112us) server_negotiation.cc:407] Connection header received
0927 22:49:15.687652 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.687658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.687763 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.687870 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:15.689259 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.689951 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.691187 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.691605 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.691696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.695424 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:15.695459 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:15.695477 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:15.695510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:15.698046 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.698664 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.698671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.698673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.698788 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.699107 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.699109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.699111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.699463 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:15.699665 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.699968 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.700141 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:49:15.700992 12039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:49:15.741545 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.741121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57670)
0927 22:49:15.741303 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.741393 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 22:49:15.742502 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.742175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57678)
0927 22:49:15.742291 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.742358 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 22:49:15.787199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 11907
I20260927 22:49:15.800099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 12050
I20260927 22:49:15.803036 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.802597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57694)
0927 22:49:15.802764 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.802855 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:49:15.803947 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.803630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57708)
0927 22:49:15.803751 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.803840 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Sep 27 22:49:15 dist-test-slave-kr25 krb5kdc[11212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:49:15.814661 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.800786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.0.97.3:40573)
0927 22:49:15.800937 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:15.800941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:15.801049 (+ 108us) server_negotiation.cc:407] Connection header received
0927 22:49:15.801243 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:15.801247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:15.801307 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:15.801386 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:15.802738 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.803738 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.805232 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:15.805600 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:15.805689 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:15.809892 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:15.809925 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:15.809942 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:15.809978 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:15.812431 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.813077 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.813084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.813086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.813206 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:15.813558 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:15.813561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:15.813563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:15.813877 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:15.814096 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:15.814273 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 22:49:15.814416 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:49:15.815200 12173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:49:15.884609 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.884128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57712)
0927 22:49:15.884339 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.884451 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:49:15.885572 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:15.885211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41627 (local address 127.0.0.1:57726)
0927 22:49:15.885386 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:49:15.885468 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:49:15.902211 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 12050
I20260927 22:49:15.914829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 11656
I20260927 22:49:15.915985 11718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:49:16.024099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 11656
I20260927 22:49:16.040534 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:49:16.376058 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:49:16.376271 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:49:16.711474 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:49:16.711 UTC [11242] LOG: received smart shutdown request
2026-09-27 22:49:16.715 UTC [11242] LOG: background worker "logical replication launcher" (PID 11251) exited with exit code 1
2026-09-27 22:49:16.715 UTC [11246] LOG: shutting down
2026-09-27 22:49:16.868 UTC [11242] LOG: database system is shut down
2026-09-27T22:49:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47832 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:49:17 dist-test-slave-kr25 krb5kdc[12238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:49:17 dist-test-slave-kr25 krb5kdc[12238](info): set up 2 sockets
Sep 27 22:49:17 dist-test-slave-kr25 krb5kdc[12238](info): commencing operation
krb5kdc: starting...
W20260927 22:49:17.995707 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:49:18 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:49:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 22:49:18.301682 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfHsC0g/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/postgres -l logfile start
2026-09-27 22:49:20.700 UTC [12268] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:49:20.700 UTC [12268] LOG: listening on IPv4 address "127.0.97.20", port 36851
2026-09-27 22:49:20.703 UTC [12268] LOG: listening on Unix socket "/tmp/.s.PGSQL.36851"
2026-09-27 22:49:20.725 UTC [12271] LOG: database system was shut down at 2026-09-27 22:49:19 UTC
2026-09-27 22:49:20.728 UTC [12268] LOG: database system is ready to accept connections
I20260927 22:49:21.611140 388 mini_postgres.cc:96] Postgres bound to 36851
2026-09-27 22:49:21.626 UTC [12279] FATAL: database "slave" does not exist
127.0.97.20:36851 - accepting connections
I20260927 22:49:21.626933 388 mini_ranger.cc:161] Starting Ranger...
I20260927 22:49:21.650897 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:49:22.107523 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:49:22.107628 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-admin
2026-09-27 22:49:22,390 [I] DB FLAVOR :POSTGRES
2026-09-27 22:49:22,391 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:49:22,391 [I] Checking connection..
2026-09-27 22:49:22,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:49:22,634 [I] Checking connection passed.
2026-09-27 22:49:22,634 [I] --------- Verifying version history table ---------
2026-09-27 22:49:22,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:49:22,877 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:49:22,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:49:23,099 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:49:23,099 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:49:23,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:49:23.343 UTC [12343] WARNING: there is no transaction in progress
2026-09-27 22:49:23,349 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:49:23,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:49:23,586 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:49:23,586 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:49:23,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:49:23,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:49:24,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kr25.c.gcp-upstream.internal','N') ;"
2026-09-27 22:49:24,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:49:24,556 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:49:24,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:49:24,793 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:49:24,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:49:25,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:49:25,287 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:49:25,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:49:26.206 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.223 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.240 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.255 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.269 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.289 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.343 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.349 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.364 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.374 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.384 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.397 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.407 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.413 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.420 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:26.427 UTC [12451] WARNING: there is no transaction in progress
2026-09-27 22:49:27,032 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:49:27,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:49:27,256 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:49:27,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:49:27,506 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:49:27,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:49:27,729 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:49:27,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:49:27,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:49:28,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kr25.c.gcp-upstream.internal';"
2026-09-27 22:49:28,450 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:49:28,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:49:28,690 [I] DB_PATCHES have already been applied
I20260927 22:49:28.695719 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfHsC0g/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:49:28.695796 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 22:49:28.708570 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:44473
Sep 27, 2026 10:49:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:49:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:49:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:49:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:49:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:49:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:49:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:49:29.736485 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.110s user 0.005s sys 0.031s
Sep 27, 2026 10:49:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:49:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:49:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:49:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:49:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:49:50.926009 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:49:51 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549391, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:49:51.087338 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:49:51.105712 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:49:51.928711 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:49:51.928824 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms
I20260927 22:49:51.928864 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:36851
2026-09-27 22:49:52,243 [I] DB FLAVOR :POSTGRES
2026-09-27 22:49:52,244 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:49:52,244 [I] Checking connection
2026-09-27 22:49:52,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:49:52,471 [I] connection success
2026-09-27 22:49:52,471 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:49:52,471 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:49:52,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:49:52,706 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:49:52,706 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:49:52,706 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:49:52,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:49:52,945 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:49:53.415817 388 mini_ranger_kms.cc:310] Created kms service
I20260927 22:49:53.613755 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:49:53.658174 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:49:53 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549391, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:49:54.055235 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:49:54.055585 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfHsC0g/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfHsC0g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:49:54.055644 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 22:49:54.072438 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12631
I20260927 22:49:54.072559 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:39067
Sep 27, 2026 10:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:49:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:49:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:49:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:49:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:49:56.150830 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.063s user 0.066s sys 0.262s
I20260927 22:49:56.151072 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:49:56.151230 388 mini_ranger_kms.cc:401] 127.0.97.20:39067/kms/v1/keys
Sep 27 22:49:56 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549396, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:49:56 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549396, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:49:57 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549396, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:49:59 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:49:59.259771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41917
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41917
--ranger_config_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:49:59.578663 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:49:59.578975 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:49:59.579102 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:49:59.586345 12682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:49:59.586457 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:49:59.586529 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:49:59.586597 12682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:49:59.586658 12682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:49:59.596784 12682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:39067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41917
--ranger_config_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:49:59.598016 12682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:49:59.599287 12682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:49:59.611886 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:49:59.611913 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:49:59.612753 12682 server_base.cc:1034] running on GCE node
W20260927 22:49:59.612903 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:49:59.613683 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:49:59.615206 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:49:59.616417 12682 hybrid_clock.cc:648] HybridClock initialized: now 1790549399616349 us; error 63 us; skew 500 ppm
I20260927 22:49:59.616746 12682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:49:59 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:49:59.623008 12682 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:49:59.624527 12682 webserver.cc:460] Webserver started at http://127.0.97.62:34699/ using document root <none> and password file <none>
I20260927 22:49:59.625145 12682 fs_manager.cc:359] Metadata directory not provided
I20260927 22:49:59.625234 12682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:49:59.625506 12682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:49:59 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:49:59.681967 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "b550c4ed6f444366a6f0ec7420c567dd"
format_stamp: "Formatted at 2026-09-27 22:49:59 on dist-test-slave-kr25"
server_key: "213051a6613cad719e9cc02f6d15a55e"
server_key_iv: "518b31d78060c2755869f5d43e8c39c4"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:59.683244 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "b550c4ed6f444366a6f0ec7420c567dd"
format_stamp: "Formatted at 2026-09-27 22:49:59 on dist-test-slave-kr25"
server_key: "213051a6613cad719e9cc02f6d15a55e"
server_key_iv: "518b31d78060c2755869f5d43e8c39c4"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:59.690143 12682 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260927 22:49:59.709339 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:49:59.711062 12682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:49:59.711243 12682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "b550c4ed6f444366a6f0ec7420c567dd"
format_stamp: "Formatted at 2026-09-27 22:49:59 on dist-test-slave-kr25"
server_key: "213051a6613cad719e9cc02f6d15a55e"
server_key_iv: "518b31d78060c2755869f5d43e8c39c4"
server_key_version: "kuduclusterkey@0"
I20260927 22:49:59.711519 12682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:49:59.743304 12682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:49:59.748150 12682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:49:59.748474 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:49:59.766664 12682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41917
I20260927 22:49:59.766669 12748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41917 every 8 connection(s)
I20260927 22:49:59.768500 12682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:49:59.772503 12749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:49:59.774451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 12682
I20260927 22:49:59.774668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:49:59.775529 388 external_mini_cluster.cc:1428] Setting key 0b1a7b8c4b16875bb4b6ea05473f8f74
I20260927 22:49:59.786145 12749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Bootstrap starting.
I20260927 22:49:59.790367 12749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Neither blocks nor log segments found. Creating new log.
I20260927 22:49:59.791851 12749 log.cc:824] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:49:59.794663 12749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: No bootstrap required, opened a new log
I20260927 22:49:59.803210 12749 raft_consensus.cc:348] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:49:59.803678 12749 raft_consensus.cc:374] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:49:59.803776 12749 raft_consensus.cc:729] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550c4ed6f444366a6f0ec7420c567dd, State: Initialized, Role: FOLLOWER
I20260927 22:49:59.804446 12749 consensus_queue.cc:260] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:49:59.804781 12749 raft_consensus.cc:388] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:49:59.804963 12749 raft_consensus.cc:482] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:49:59.805163 12749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 0 FOLLOWER]: Advancing to term 1
Sep 27 22:49:59 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:49:59.807916 12749 raft_consensus.cc:504] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:49:59.808452 12749 leader_election.cc:302] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b550c4ed6f444366a6f0ec7420c567dd; no voters:
I20260927 22:49:59.809093 12749 leader_election.cc:288] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:49:59.809513 12754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:49:59.810393 12754 raft_consensus.cc:686] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 LEADER]: Becoming Leader. State: Replica: b550c4ed6f444366a6f0ec7420c567dd, State: Running, Role: LEADER
I20260927 22:49:59.811121 12754 consensus_queue.cc:237] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:49:59.813170 12749 sys_catalog.cc:564] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:49:59.813968 12756 sys_catalog.cc:455] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b550c4ed6f444366a6f0ec7420c567dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } } }
I20260927 22:49:59.814316 12756 sys_catalog.cc:458] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: This master's current role is: LEADER
I20260927 22:49:59.816325 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:49:59.777728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.0.97.62:41917)
0927 22:49:59.792334 (+ 14606us) server_negotiation.cc:206] Beginning negotiation
0927 22:49:59.792350 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:49:59.792398 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:49:59.794811 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:49:59.794859 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:49:59.795367 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:49:59.795808 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:49:59.797321 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:59.799413 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:59.801228 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:49:59.802322 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:49:59.802495 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:49:59.807162 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:49:59.807174 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:49:59.807194 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:49:59.807240 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:49:59.811213 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:59.812039 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:59.812048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:59.812057 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:59.812214 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:49:59.812584 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:49:59.812589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:49:59.812593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:49:59.813356 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:49:59.813639 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:49:59.814970 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0927 22:49:59.815272 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14351,"thread_start_us":14798,"threads_started":1}
I20260927 22:49:59.821439 12755 sys_catalog.cc:455] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b550c4ed6f444366a6f0ec7420c567dd. Latest consensus state: current_term: 1 leader_uuid: "b550c4ed6f444366a6f0ec7420c567dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } } }
I20260927 22:49:59.821731 12755 sys_catalog.cc:458] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: This master's current role is: LEADER
I20260927 22:49:59.822494 12759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:49:59.826922 12759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:49:59.829290 12749 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:49:59.842821 12759 catalog_manager.cc:1347] Generated new cluster ID: f55313ec2bfc4f23b0a099afb074b6b9
I20260927 22:49:59.842962 12759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:49:59.865628 12759 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:49:59.867962 12759 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:49:59.882611 12759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Generated new TSK 0
I20260927 22:49:59.883728 12759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:50:00.639472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:50:00 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:50:01.056692 12794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:01.057149 12794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:01.057360 12794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:01.068562 12794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:01.068709 12794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:01.068951 12794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:50:01.084447 12794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:01.086324 12794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:01.088090 12794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:01.104316 12804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:01.106094 12805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:01.106747 12807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:01.107357 12794 server_base.cc:1034] running on GCE node
I20260927 22:50:01.108131 12794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:01.109337 12794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:01.110594 12794 hybrid_clock.cc:648] HybridClock initialized: now 1790549401110530 us; error 48 us; skew 500 ppm
I20260927 22:50:01.110978 12794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:01 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:01.114858 12794 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:50:01.118072 12794 webserver.cc:460] Webserver started at http://127.0.97.1:37773/ using document root <none> and password file <none>
I20260927 22:50:01.118974 12794 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:01.119117 12794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:01.119499 12794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:50:01 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:01.152531 12794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "b6945c336cb641669efe129b6ec2a784"
format_stamp: "Formatted at 2026-09-27 22:50:01 on dist-test-slave-kr25"
server_key: "18febe3f9f503ce9097e1a4f149192de"
server_key_iv: "a738051197fada94e242581cac87423a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:01.153736 12794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b6945c336cb641669efe129b6ec2a784"
format_stamp: "Formatted at 2026-09-27 22:50:01 on dist-test-slave-kr25"
server_key: "18febe3f9f503ce9097e1a4f149192de"
server_key_iv: "a738051197fada94e242581cac87423a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:01.167574 12794 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.008s
I20260927 22:50:01.188045 12814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:01.189864 12794 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:50:01.190124 12794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "b6945c336cb641669efe129b6ec2a784"
format_stamp: "Formatted at 2026-09-27 22:50:01 on dist-test-slave-kr25"
server_key: "18febe3f9f503ce9097e1a4f149192de"
server_key_iv: "a738051197fada94e242581cac87423a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:01.190605 12794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:01.236126 12794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:01.241210 12794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:01.241524 12794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:01.242758 12794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:01.244519 12794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:01.244614 12794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:01.244781 12794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:01.244864 12794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:01.329589 12794 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33487
I20260927 22:50:01.329604 12926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33487 every 8 connection(s)
I20260927 22:50:01.331462 12794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:50:01.334717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 12794
I20260927 22:50:01.334961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:50:01.335978 388 external_mini_cluster.cc:1428] Setting key 32d49415b57a16c3235430653ebbb8f4
Sep 27 22:50:01 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 27 22:50:01 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:01.423491 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:01.347318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59407 (local address 127.0.97.62:41917)
0927 22:50:01.371877 (+ 24559us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:01.371884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:01.375596 (+ 3712us) server_negotiation.cc:407] Connection header received
0927 22:50:01.388187 (+ 12591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:01.388196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:01.388329 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:01.388420 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:01.390907 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:01.392114 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:01.399479 (+ 7365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:01.400156 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:01.400264 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:01.415335 (+ 15071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:01.415345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:01.415350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:01.415404 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:01.419455 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:01.420531 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:01.420538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:01.420542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:01.420702 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:01.421281 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:01.421287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:01.421291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:01.421649 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:01.421764 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:01.422836 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0927 22:50:01.423059 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24386,"thread_start_us":24577,"threads_started":1}
I20260927 22:50:01.425757 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:01.346693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.1:59407)
0927 22:50:01.372650 (+ 25957us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:01.372713 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:01.387731 (+ 15018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:01.389112 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:01.389126 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:01.389861 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:01.390725 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:01.390752 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:01.395096 (+ 4344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:01.395103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:01.398894 (+ 3791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:01.398915 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:01.401466 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:01.401474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:01.401772 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:01.406952 (+ 5180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:01.406992 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:01.415000 (+ 8008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:01.419684 (+ 4684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:01.419696 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:01.419717 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:01.420377 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:01.420860 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:01.420865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:01.420869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:01.421124 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:01.421880 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:01.421889 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:01.422393 (+ 504us) client_negotiation.cc:769] Sending connection context
0927 22:50:01.424233 (+ 1840us) client_negotiation.cc:240] Negotiation successful
0927 22:50:01.424722 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25566,"thread_start_us":32639,"threads_started":1}
I20260927 22:50:01.427580 12927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:50:01.427893 12927 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:01.428663 12927 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:50:01.430933 12714 ts_manager.cc:194] Registered new tserver with Master: b6945c336cb641669efe129b6ec2a784 (127.0.97.1:33487)
I20260927 22:50:01.432547 12714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260927 22:50:01.448428 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.109s user 0.001s sys 0.017s
I20260927 22:50:01.449156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:50:01 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 22:50:01.870904 12933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:01.871417 12933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:01.871645 12933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:01.882882 12933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:01.883102 12933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:01.883343 12933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:50:01.898741 12933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:01.900699 12933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:01.902411 12933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:01.932576 12941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:01.941193 12940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:01.941643 12933 server_base.cc:1034] running on GCE node
W20260927 22:50:01.948004 12944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:01.949026 12933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:01.950385 12933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:01.951607 12933 hybrid_clock.cc:648] HybridClock initialized: now 1790549401951535 us; error 62 us; skew 500 ppm
I20260927 22:50:01.952030 12933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:01 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:01.956337 12933 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:50:01.981482 12933 webserver.cc:460] Webserver started at http://127.0.97.2:44219/ using document root <none> and password file <none>
I20260927 22:50:01.982656 12933 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:01.982931 12933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:01.983398 12933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:50:01 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:02.017580 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "f831a47c369a46babeaa1b65614bd44f"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "0b478651162eac9621e8d8ba55b62b92"
server_key_iv: "33c53ebe0fa66b801f8d82c61e5cd973"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:02.018788 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f831a47c369a46babeaa1b65614bd44f"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "0b478651162eac9621e8d8ba55b62b92"
server_key_iv: "33c53ebe0fa66b801f8d82c61e5cd973"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:02.032573 12933 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260927 22:50:02.076792 12957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:02.078384 12933 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260927 22:50:02.078557 12933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "f831a47c369a46babeaa1b65614bd44f"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "0b478651162eac9621e8d8ba55b62b92"
server_key_iv: "33c53ebe0fa66b801f8d82c61e5cd973"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:02.078784 12933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:02.095386 12933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:02.099336 12933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:02.099711 12933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:02.100689 12933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:02.102018 12933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:02.102093 12933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:02.102234 12933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:02.102290 12933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:02.138742 12933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34179
I20260927 22:50:02.138746 13069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34179 every 8 connection(s)
I20260927 22:50:02.140233 12933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:50:02.144652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 12933
I20260927 22:50:02.144825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:50:02.145412 388 external_mini_cluster.cc:1428] Setting key 216dac7b3c0486bc0bc2f2907f9c01b8
Sep 27 22:50:02 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:02.168085 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:02.146190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59721 (local address 127.0.97.62:41917)
0927 22:50:02.151102 (+ 4912us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:02.151108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:02.154295 (+ 3187us) server_negotiation.cc:407] Connection header received
0927 22:50:02.154373 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:02.154378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:02.154456 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:02.154557 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:02.156336 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.157073 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.158617 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.159047 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.159251 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:02.162624 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:02.162630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:02.162633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:02.162667 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:02.165104 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.165778 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.165783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.165785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.165920 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.166386 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.166390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.166392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.166656 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:02.166784 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:02.167507 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 22:50:02.167734 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4771,"thread_start_us":4653,"threads_started":1}
I20260927 22:50:02.169269 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:02.143778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.2:59721)
0927 22:50:02.151690 (+ 7912us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:02.151754 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:02.153919 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:02.154843 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:02.154855 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:02.155475 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:02.156155 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:02.156174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.157223 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.157227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:02.158472 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:02.158485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.159325 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.159333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:02.159609 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:02.160231 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:02.160254 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:02.162417 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:02.165241 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:02.165251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:02.165269 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:02.165658 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:02.166020 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:02.166025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:02.166029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:02.166268 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:02.166774 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:02.166782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:02.167115 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 22:50:02.167344 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 22:50:02.167908 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7571,"thread_start_us":7328,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:50:02.170753 13070 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:50:02.171042 13070 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:02.171873 13070 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:50:02.173408 12714 ts_manager.cc:194] Registered new tserver with Master: f831a47c369a46babeaa1b65614bd44f (127.0.97.2:34179)
I20260927 22:50:02.174448 12714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:50:02.202608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:50:02.435956 12927 heartbeater.cc:503] Master 127.0.97.62:41917 was elected leader, sending a full tablet report...
W20260927 22:50:02.522128 13076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:02.522405 13076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:02.522539 13076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:02.529716 13076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:02.529852 13076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:02.530006 13076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:50:02.539695 13076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:02.540923 13076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:02.542126 13076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:02.553921 13081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:02.553962 13084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:02.553990 13082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:02.554566 13076 server_base.cc:1034] running on GCE node
I20260927 22:50:02.555099 13076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:02.555961 13076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:02.557142 13076 hybrid_clock.cc:648] HybridClock initialized: now 1790549402557109 us; error 45 us; skew 500 ppm
I20260927 22:50:02.557412 13076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:02 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:02.560272 13076 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:50:02.561573 13076 webserver.cc:460] Webserver started at http://127.0.97.3:32873/ using document root <none> and password file <none>
I20260927 22:50:02.562136 13076 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:02.562227 13076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:02.562466 13076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:50:02 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:02.579377 13076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "49589ba99da1417a99454dbb9a74b456"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "584cb849981b9b55d13ab2157e24992e"
server_key_iv: "3a79b03d86a31fb2966dbcc7d8c4ca0a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:02.580535 13076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "49589ba99da1417a99454dbb9a74b456"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "584cb849981b9b55d13ab2157e24992e"
server_key_iv: "3a79b03d86a31fb2966dbcc7d8c4ca0a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:02.586061 13076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:50:02.601655 13091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:02.603117 13076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:50:02.603253 13076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "49589ba99da1417a99454dbb9a74b456"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "584cb849981b9b55d13ab2157e24992e"
server_key_iv: "3a79b03d86a31fb2966dbcc7d8c4ca0a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:02.603466 13076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:02.615702 13076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:02.619222 13076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:02.619500 13076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:02.620378 13076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:02.621603 13076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:02.621670 13076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:02.621768 13076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:02.621822 13076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:02.650067 13076 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36773
I20260927 22:50:02.650120 13203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36773 every 8 connection(s)
I20260927 22:50:02.651321 13076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:50:02.656157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13076
I20260927 22:50:02.656306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:50:02.656759 388 external_mini_cluster.cc:1428] Setting key 72669263b231b17ffb10983f540eb304
Sep 27 22:50:02 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:02.671756 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:02.653870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52419 (local address 127.0.97.62:41917)
0927 22:50:02.654086 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:02.654093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:02.655313 (+ 1220us) server_negotiation.cc:407] Connection header received
0927 22:50:02.657164 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:02.657172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:02.657289 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:02.657395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:02.659007 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.660068 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.661605 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.662032 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.662134 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:02.665226 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:02.665233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:02.665239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:02.665281 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:02.668479 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.669176 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.669182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.669186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.669331 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.669792 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.669798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.669802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.670172 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:02.670319 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:02.671152 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 22:50:02.671396 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:50:02.672706 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:02.654428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.3:52419)
0927 22:50:02.655168 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:02.655206 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:02.656905 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:02.657655 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:02.657662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:02.658206 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:02.658844 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:02.658869 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.660234 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.660238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:02.661469 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:02.661480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.662196 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.662200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:02.662423 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:02.663063 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:02.663086 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:02.665010 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:02.668597 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:02.668606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:02.668622 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:02.669071 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:02.669428 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:02.669433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:02.669437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:02.669674 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:02.670394 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:02.670401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:02.670764 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 22:50:02.671074 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 22:50:02.671571 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":324,"threads_started":1}
I20260927 22:50:02.673822 13204 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:50:02.674090 13204 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:02.674783 13204 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:50:02.676105 12714 ts_manager.cc:194] Registered new tserver with Master: 49589ba99da1417a99454dbb9a74b456 (127.0.97.3:36773)
I20260927 22:50:02.676889 12714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52419
I20260927 22:50:02.684269 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:50:02.686292 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 12794
Sep 27 22:50:02 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:50:02.706066 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:02.687528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.0.97.1:33487)
0927 22:50:02.688132 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:02.688141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:02.688172 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:50:02.688247 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:02.688256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:02.688541 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:02.688714 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:02.690405 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.691986 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.693605 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.694413 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.694531 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:02.699914 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:02.699925 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:02.699939 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:02.699974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:02.703128 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.703962 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.703969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.703971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.704107 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.704484 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.704487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.704489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.704786 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:02.705010 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:02.705485 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 22:50:02.705726 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":321,"threads_started":1}
I20260927 22:50:02.707368 12922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:02.793717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 12794
I20260927 22:50:02.805931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 12933
Sep 27 22:50:02 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:50:02.821889 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:02.806818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.97.2:34179)
0927 22:50:02.807317 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:02.807323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:02.807356 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:50:02.807416 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:02.807423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:02.807676 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:02.807816 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:02.809005 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.809969 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.811285 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.812147 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.812259 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:02.816836 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:02.816847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:02.816863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:02.816904 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:02.819309 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.819960 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.819967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.819969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.820101 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.820429 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.820433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.820435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.820754 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:02.821006 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:02.821315 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 22:50:02.821576 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260927 22:50:02.823016 13065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:02.903069 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 12933
I20260927 22:50:02.915074 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13076
Sep 27 22:50:02 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:50:02.931036 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:02.916070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.0.97.3:36773)
0927 22:50:02.916715 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:02.916721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:02.916750 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:50:02.916840 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:02.916846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:02.917050 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:02.917199 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:02.918458 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.919425 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.920737 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:02.921269 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:02.921360 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:02.925652 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:02.925662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:02.925676 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:02.925721 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:02.928534 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.929173 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.929179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.929182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.929317 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:02.929682 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:02.929685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:02.929687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:02.930017 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:02.930244 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:02.930549 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 22:50:02.930755 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":362,"threads_started":1}
I20260927 22:50:02.932332 13199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:03.013427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13076
I20260927 22:50:03.025593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 12682
I20260927 22:50:03.026706 12744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:03.128576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 12682
I20260927 22:50:03.146668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41917
--webserver_interface=127.0.97.62
--webserver_port=34699
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41917
--ranger_config_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:03.452483 13215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:03.452768 13215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:03.452874 13215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:03.460016 13215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:03.460116 13215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:03.460158 13215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:03.460211 13215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:03.460258 13215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:03.470028 13215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41917
--ranger_config_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:03.471374 13215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:03.472713 13215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:03.483745 13220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:03.483697 13223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:03.483786 13221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:03.484326 13215 server_base.cc:1034] running on GCE node
I20260927 22:50:03.485044 13215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:03.486480 13215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:03.487682 13215 hybrid_clock.cc:648] HybridClock initialized: now 1790549403487626 us; error 47 us; skew 500 ppm
I20260927 22:50:03.488013 13215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:03 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:03.490978 13215 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:50:03.492410 13215 webserver.cc:460] Webserver started at http://127.0.97.62:34699/ using document root <none> and password file <none>
I20260927 22:50:03.493014 13215 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:03.493113 13215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:50:03 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:03.556838 13215 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:50:03.560369 13231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:03.561902 13215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:50:03.562094 13215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "b550c4ed6f444366a6f0ec7420c567dd"
format_stamp: "Formatted at 2026-09-27 22:49:59 on dist-test-slave-kr25"
server_key: "213051a6613cad719e9cc02f6d15a55e"
server_key_iv: "518b31d78060c2755869f5d43e8c39c4"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:03.563079 13215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:03.586184 13215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:03.590097 13215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:03.590394 13215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:03.608274 13215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41917
I20260927 22:50:03.608299 13282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41917 every 8 connection(s)
I20260927 22:50:03.609795 13215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:50:03.616592 13283 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:50:03.617405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13215
I20260927 22:50:03.618383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33487
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41917
--webserver_port=37773
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:50:03.621127 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Bootstrap starting.
I20260927 22:50:03.640182 13283 log.cc:824] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:03.648581 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:50:03.649060 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Bootstrap complete.
I20260927 22:50:03.657603 13283 raft_consensus.cc:348] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:50:03.658439 13283 raft_consensus.cc:729] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550c4ed6f444366a6f0ec7420c567dd, State: Initialized, Role: FOLLOWER
I20260927 22:50:03.659010 13283 consensus_queue.cc:260] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:50:03.659211 13283 raft_consensus.cc:388] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:50:03.659322 13283 raft_consensus.cc:482] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:50:03.659493 13283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:50:03.663259 13283 raft_consensus.cc:504] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:50:03.663714 13283 leader_election.cc:302] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b550c4ed6f444366a6f0ec7420c567dd; no voters:
I20260927 22:50:03.664361 13283 leader_election.cc:288] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:50:03.664494 13287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:50:03.666402 13283 sys_catalog.cc:564] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:03.671334 13287 raft_consensus.cc:686] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [term 2 LEADER]: Becoming Leader. State: Replica: b550c4ed6f444366a6f0ec7420c567dd, State: Running, Role: LEADER
I20260927 22:50:03.671967 13287 consensus_queue.cc:237] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:50:03.674381 13288 sys_catalog.cc:455] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b550c4ed6f444366a6f0ec7420c567dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } } }
I20260927 22:50:03.674737 13288 sys_catalog.cc:458] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:03.675251 13289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b550c4ed6f444366a6f0ec7420c567dd. Latest consensus state: current_term: 2 leader_uuid: "b550c4ed6f444366a6f0ec7420c567dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b550c4ed6f444366a6f0ec7420c567dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } } }
I20260927 22:50:03.675532 13289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:03.684027 13298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:50:03.689302 13298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:50:03.690024 13283 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:50:03.690665 13298 catalog_manager.cc:1259] Loaded cluster ID: f55313ec2bfc4f23b0a099afb074b6b9
I20260927 22:50:03.690785 13298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:50:03.694653 13298 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:50:03.696049 13298 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Loaded TSK: 0
I20260927 22:50:03.697489 13298 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:50:03.991451 13285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:03.991846 13285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:03.992089 13285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:04.003330 13285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:04.003644 13285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:04.003984 13285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:50:04.019728 13285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37773
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:04.021663 13285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:04.023372 13285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:04.043732 13321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:04.045554 13322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:04.050289 13285 server_base.cc:1034] running on GCE node
W20260927 22:50:04.050599 13324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:04.053370 13285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:04.054817 13285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:04.058184 13285 hybrid_clock.cc:648] HybridClock initialized: now 1790549404058071 us; error 323 us; skew 500 ppm
I20260927 22:50:04.058593 13285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:04 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:04.063521 13285 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:50:04.081594 13285 webserver.cc:460] Webserver started at http://127.0.97.1:37773/ using document root <none> and password file <none>
I20260927 22:50:04.082638 13285 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:04.082818 13285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:50:04 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:04.113057 13285 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:50:04.117347 13332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:04.119515 13285 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260927 22:50:04.119692 13285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "b6945c336cb641669efe129b6ec2a784"
format_stamp: "Formatted at 2026-09-27 22:50:01 on dist-test-slave-kr25"
server_key: "18febe3f9f503ce9097e1a4f149192de"
server_key_iv: "a738051197fada94e242581cac87423a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:04.121141 13285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:04.154903 13285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:04.160223 13285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:04.160542 13285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:04.166579 13285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:04.173260 13285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:04.173403 13285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:04.173543 13285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:04.173624 13285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:04.280220 13285 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33487
I20260927 22:50:04.280290 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33487 every 8 connection(s)
I20260927 22:50:04.282045 13285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:50:04.283866 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13285
I20260927 22:50:04.284634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34179
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41917
--webserver_port=44219
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:50:04 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:04.330183 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:04.289247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.1:60585)
0927 22:50:04.293518 (+ 4271us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:04.293576 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:04.297995 (+ 4419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:04.300448 (+ 2453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:04.300459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:04.301125 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:04.301878 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:04.301903 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.312506 (+ 10603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.312512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:04.314902 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:04.314921 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.316938 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.316947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:04.317249 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:04.318040 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:04.318071 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:04.320770 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:04.325142 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:04.325151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:04.325168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:04.325643 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:04.326262 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:04.326267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:04.326271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:04.326577 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:04.327519 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:04.327528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:04.327978 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 22:50:04.328253 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 22:50:04.328812 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3952,"thread_start_us":3711,"threads_started":1}
I20260927 22:50:04.330184 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:04.289244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60585 (local address 127.0.97.62:41917)
0927 22:50:04.297264 (+ 8020us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:04.297285 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:04.297343 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:50:04.298845 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:04.298864 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:04.299370 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:04.299775 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:04.302100 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.312239 (+ 10139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.315143 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.316313 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.316513 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:04.321145 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:04.321157 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:04.321180 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:04.321234 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:04.324968 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:04.325816 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:04.325825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:04.325834 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:04.326039 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:04.326718 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:04.326723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:04.326728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:04.327354 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:04.327596 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:04.328500 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 22:50:04.328864 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7743,"thread_start_us":7520,"threads_started":1}
I20260927 22:50:04.334280 13446 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:50:04.334661 13446 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:04.335642 13446 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
W20260927 22:50:04.338851 13446 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41917 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:50:04.339082 13446 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:04.339363 13446 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:50:04.340555 13446 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:04.340816 13446 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
W20260927 22:50:04.341833 13446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:50:04.683296 13450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:04.683597 13450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:04.683738 13450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:04.690940 13450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:04.691042 13450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:04.691186 13450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:50:04.702944 13450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44219
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:04.704619 13450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:04.706171 13450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:04.720482 13464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:04.721078 13465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:04.725056 13467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:04.725468 13450 server_base.cc:1034] running on GCE node
I20260927 22:50:04.726235 13450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:04.727481 13450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:04.730849 13450 hybrid_clock.cc:648] HybridClock initialized: now 1790549404730786 us; error 46 us; skew 500 ppm
I20260927 22:50:04.731187 13450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:04 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:04.738240 13450 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:50:04.740012 13450 webserver.cc:460] Webserver started at http://127.0.97.2:44219/ using document root <none> and password file <none>
I20260927 22:50:04.740707 13450 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:04.740835 13450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:50:04 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:04.770052 13450 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.006s sys 0.005s
I20260927 22:50:04.784121 13475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:04.788873 13450 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.002s
I20260927 22:50:04.789179 13450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "f831a47c369a46babeaa1b65614bd44f"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "0b478651162eac9621e8d8ba55b62b92"
server_key_iv: "33c53ebe0fa66b801f8d82c61e5cd973"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:04.790763 13450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:04.814324 13450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:04.820019 13450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:04.820393 13450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:04.826751 13450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:04.828394 13450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:04.828487 13450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:04.828594 13450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:04.828667 13450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:04.868408 13450 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34179
I20260927 22:50:04.868458 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34179 every 8 connection(s)
I20260927 22:50:04.869793 13450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:50:04.872174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13450
I20260927 22:50:04.872911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36773
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41917
--webserver_port=32873
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:50:04 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:50:04 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:04.908995 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:04.872612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54707 (local address 127.0.97.62:41917)
0927 22:50:04.887073 (+ 14461us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:04.887080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:04.887105 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:50:04.887174 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:04.887180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:04.887264 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:04.887382 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:04.889583 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.890755 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.893097 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.893793 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.893944 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:04.897814 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:04.897821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:04.897826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:04.897865 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:04.905450 (+ 7585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:04.906348 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:04.906354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:04.906358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:04.906510 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:04.906985 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:04.906990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:04.906994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:04.907316 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:04.907482 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:04.908388 (+ 906us) server_negotiation.cc:299] Negotiation successful
0927 22:50:04.908627 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14325,"thread_start_us":14206,"threads_started":1}
I20260927 22:50:04.914728 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:04.873319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.2:54707)
0927 22:50:04.884105 (+ 10786us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:04.884162 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:04.886668 (+ 2506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:04.887877 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:04.887890 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:04.888504 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:04.889357 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:04.889385 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.890954 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.890959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:04.892926 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:04.892942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:04.894060 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:04.894067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:04.894346 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:04.895146 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:04.895171 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:04.897552 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:04.905611 (+ 8059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:04.905620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:04.905639 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:04.906210 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:04.906626 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:04.906631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:04.906635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:04.906849 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:04.907552 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:04.907561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:04.907973 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 22:50:04.908260 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 22:50:04.908795 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10465,"thread_start_us":10240,"threads_started":1}
I20260927 22:50:04.916556 13590 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:50:04.916893 13590 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:04.917968 13590 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:50:04.920188 13248 ts_manager.cc:194] Registered new tserver with Master: f831a47c369a46babeaa1b65614bd44f (127.0.97.2:34179)
I20260927 22:50:04.922374 13248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54707
Sep 27 22:50:05 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 22:50:05.340500 13594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:05.340901 13594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:05.341064 13594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:50:05.341972 13446 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:05.342357 13446 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:50:05.343564 13248 ts_manager.cc:194] Registered new tserver with Master: b6945c336cb641669efe129b6ec2a784 (127.0.97.1:33487)
I20260927 22:50:05.344645 13248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60585
W20260927 22:50:05.351722 13594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:05.351845 13594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:05.352036 13594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:50:05.367081 13594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32873
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:05.368781 13594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:05.370473 13594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:05.387902 13600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:05.389762 13601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:05.398408 13594 server_base.cc:1034] running on GCE node
W20260927 22:50:05.406891 13603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:05.407979 13594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:05.410115 13594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:05.411468 13594 hybrid_clock.cc:648] HybridClock initialized: now 1790549405411380 us; error 54 us; skew 500 ppm
I20260927 22:50:05.411831 13594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:05 dist-test-slave-kr25 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:05.415096 13594 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:50:05.416980 13594 webserver.cc:460] Webserver started at http://127.0.97.3:32873/ using document root <none> and password file <none>
I20260927 22:50:05.417688 13594 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:05.417817 13594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:50:05 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:05.442500 13594 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:50:05.445377 13611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:05.446743 13594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:50:05.446904 13594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "49589ba99da1417a99454dbb9a74b456"
format_stamp: "Formatted at 2026-09-27 22:50:02 on dist-test-slave-kr25"
server_key: "584cb849981b9b55d13ab2157e24992e"
server_key_iv: "3a79b03d86a31fb2966dbcc7d8c4ca0a"
server_key_version: "kuduclusterkey@0"
I20260927 22:50:05.447907 13594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:05.466285 13594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:05.470217 13594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:05.470474 13594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:05.471381 13594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:05.472826 13594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:05.472915 13594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:05.473017 13594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:05.473088 13594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:05.514400 13594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36773
I20260927 22:50:05.514447 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36773 every 8 connection(s)
I20260927 22:50:05.516305 13594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:50:05.519393 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13594
Sep 27 22:50:05 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:05.563858 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.520408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.0.97.62:41917)
0927 22:50:05.539977 (+ 19569us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.539985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.543521 (+ 3536us) server_negotiation.cc:407] Connection header received
0927 22:50:05.543592 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.543598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.543684 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.543785 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:05.546570 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.547665 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.550951 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.551577 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.551694 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.555667 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:05.555674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:05.555679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:05.555715 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:05.560040 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:05.560879 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:05.560885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:05.560889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:05.561041 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:05.561480 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:05.561486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:05.561490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:05.561828 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:05.562074 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.563213 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.563424 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19404,"thread_start_us":19575,"threads_started":1}
I20260927 22:50:05.564174 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.540173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38295 (local address 127.0.97.62:41917)
0927 22:50:05.540653 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.540658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.542056 (+ 1398us) server_negotiation.cc:407] Connection header received
0927 22:50:05.544507 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.544513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.544595 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.544693 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:05.547779 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.548955 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.553099 (+ 4144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.553657 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.553735 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.557684 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:05.557690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:05.557694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:05.557743 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:05.560901 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:05.561696 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:05.561701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:05.561704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:05.561838 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:05.562274 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:05.562279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:05.562282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:05.562579 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:05.562686 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.563599 (+ 913us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.563776 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"spinlock_wait_cycles":2944,"thread_start_us":477,"threads_started":1}
I20260927 22:50:05.566223 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.522026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.3:38295)
0927 22:50:05.541444 (+ 19418us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:05.541510 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:05.544144 (+ 2634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:05.545137 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:05.545150 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:05.545719 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:05.547530 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.547561 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.550888 (+ 3327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.550893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.552952 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.552966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.554045 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.554051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.554326 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.555028 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:05.555055 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:05.557403 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:05.561039 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:05.561048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:05.561064 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:05.561579 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:05.561959 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:05.561963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:05.561966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:05.562164 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:05.562771 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:05.562779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:05.563146 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 22:50:05.564340 (+ 1194us) client_negotiation.cc:240] Negotiation successful
0927 22:50:05.564865 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19080,"thread_start_us":18865,"threads_started":1}
I20260927 22:50:05.567641 13724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:50:05.567926 13724 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:05.568744 13724 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:50:05.571528 13248 ts_manager.cc:194] Registered new tserver with Master: 49589ba99da1417a99454dbb9a74b456 (127.0.97.3:36773)
I20260927 22:50:05.572657 13248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38295
I20260927 22:50:05.579293 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:50:05.606524 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.584608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.0.97.62:41917)
0927 22:50:05.584917 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.584923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.585225 (+ 302us) server_negotiation.cc:407] Connection header received
0927 22:50:05.585510 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.585516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.585610 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.585682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:05.587846 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.588899 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.590889 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.591541 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.591641 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.598758 (+ 7117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:05.598766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:05.598772 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:05.598818 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:05.603330 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:05.604146 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:05.604152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:05.604156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:05.604311 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:05.604737 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:05.604743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:05.604746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:05.605152 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:05.605409 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.605793 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.606023 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
Sep 27 22:50:05 dist-test-slave-kr25 krb5kdc[12238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:50:05.612963 13248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:50:05.627166 13248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:50:05.712249 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.675643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.97.2:34179)
0927 22:50:05.688333 (+ 12690us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.688342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.689777 (+ 1435us) server_negotiation.cc:407] Connection header received
0927 22:50:05.689889 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.689898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.690172 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.690359 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:05.704488 (+ 14129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.706135 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.709069 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.710755 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.710866 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.711075 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.711781 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.711956 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12508,"thread_start_us":12705,"threads_started":1}
I20260927 22:50:05.713021 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.675073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34179 (local address 127.0.0.1:58652)
0927 22:50:05.689344 (+ 14271us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:05.689389 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:05.689581 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:05.690624 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:05.690633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:05.690664 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:05.704100 (+ 13436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.704122 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.706348 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.706355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.708836 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.708865 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.711268 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.711274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.711504 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.711558 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:50:05.712487 (+ 929us) client_negotiation.cc:240] Negotiation successful
0927 22:50:05.712711 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14005,"threads_started":1}
I20260927 22:50:05.722038 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.675611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.0.97.1:33487)
0927 22:50:05.684185 (+ 8574us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.684196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.713642 (+ 29446us) server_negotiation.cc:407] Connection header received
0927 22:50:05.713794 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.713802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.714069 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.714262 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:05.714988 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.716525 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.719204 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.720633 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.720725 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.720945 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.721568 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.721717 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8403,"thread_start_us":8254,"threads_started":1}
I20260927 22:50:05.722656 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.675113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33487 (local address 127.0.0.1:35786)
0927 22:50:05.713491 (+ 38378us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:05.713510 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:05.713649 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:05.714499 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:05.714503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:05.714528 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:05.714800 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.714809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.717010 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.717015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.719010 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.719027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.721138 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.721144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.721369 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.721399 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:50:05.722221 (+ 822us) client_negotiation.cc:240] Negotiation successful
0927 22:50:05.722369 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37109,"thread_start_us":37473,"threads_started":1}
I20260927 22:50:05.722754 13525 tablet_service.cc:1467] Processing CreateTablet for tablet d77d6e1a93fb416a8c1c5050d8df2f73 (DEFAULT_TABLE table=test-table [id=4dbbe683cb914742a2876a92d841ef48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:05.724227 13525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77d6e1a93fb416a8c1c5050d8df2f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:05.726244 13381 tablet_service.cc:1467] Processing CreateTablet for tablet d77d6e1a93fb416a8c1c5050d8df2f73 (DEFAULT_TABLE table=test-table [id=4dbbe683cb914742a2876a92d841ef48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:05.727869 13381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77d6e1a93fb416a8c1c5050d8df2f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:05.737303 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.688617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36773 (local address 127.0.0.1:33114)
0927 22:50:05.713214 (+ 24597us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:05.713232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:05.713350 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:05.719004 (+ 5654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:05.719008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:05.719033 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:05.719183 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.719192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.727123 (+ 7931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.727129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.728910 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.728926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.736425 (+ 7499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.736433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.736654 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.736698 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:50:05.736786 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:50:05.736978 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24501,"thread_start_us":332,"threads_started":1}
I20260927 22:50:05.740016 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.688818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.0.97.3:36773)
0927 22:50:05.717486 (+ 28668us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.717495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.717530 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:50:05.717624 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.717632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.717895 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.719351 (+ 1456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:05.719889 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.724826 (+ 4937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.729113 (+ 4287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.730537 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.739073 (+ 8536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.739372 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.739482 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.739668 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28481,"thread_start_us":30619,"threads_started":1}
I20260927 22:50:05.744012 13659 tablet_service.cc:1467] Processing CreateTablet for tablet d77d6e1a93fb416a8c1c5050d8df2f73 (DEFAULT_TABLE table=test-table [id=4dbbe683cb914742a2876a92d841ef48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:05.745508 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77d6e1a93fb416a8c1c5050d8df2f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:05.752763 13746 tablet_bootstrap.cc:492] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Bootstrap starting.
I20260927 22:50:05.755836 13746 tablet_bootstrap.cc:654] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:05.757061 13746 log.cc:824] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:05.763695 13746 tablet_bootstrap.cc:492] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: No bootstrap required, opened a new log
I20260927 22:50:05.763976 13746 ts_tablet_manager.cc:1397] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260927 22:50:05.772827 13748 tablet_bootstrap.cc:492] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Bootstrap starting.
I20260927 22:50:05.774593 13746 raft_consensus.cc:348] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.775100 13746 raft_consensus.cc:374] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:05.775204 13746 raft_consensus.cc:729] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6945c336cb641669efe129b6ec2a784, State: Initialized, Role: FOLLOWER
I20260927 22:50:05.775969 13746 consensus_queue.cc:260] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.778383 13748 tablet_bootstrap.cc:654] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:05.780325 13748 log.cc:824] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:05.786012 13748 tablet_bootstrap.cc:492] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: No bootstrap required, opened a new log
I20260927 22:50:05.786311 13748 ts_tablet_manager.cc:1397] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.007s
I20260927 22:50:05.789495 13749 tablet_bootstrap.cc:492] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Bootstrap starting.
I20260927 22:50:05.797330 13446 heartbeater.cc:503] Master 127.0.97.62:41917 was elected leader, sending a full tablet report...
I20260927 22:50:05.797446 13746 ts_tablet_manager.cc:1428] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Time spent starting tablet: real 0.033s user 0.021s sys 0.011s
I20260927 22:50:05.799742 13749 tablet_bootstrap.cc:654] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:05.801070 13749 log.cc:824] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Log is configured to *not* fsync() on all Append() calls
W20260927 22:50:05.802639 13447 tablet.cc:2369] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:05.809831 13748 raft_consensus.cc:348] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.810199 13748 raft_consensus.cc:374] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:05.810297 13748 raft_consensus.cc:729] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f831a47c369a46babeaa1b65614bd44f, State: Initialized, Role: FOLLOWER
I20260927 22:50:05.810913 13748 consensus_queue.cc:260] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.814484 13590 heartbeater.cc:503] Master 127.0.97.62:41917 was elected leader, sending a full tablet report...
I20260927 22:50:05.814522 13748 ts_tablet_manager.cc:1428] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Time spent starting tablet: real 0.028s user 0.000s sys 0.015s
I20260927 22:50:05.815100 13749 tablet_bootstrap.cc:492] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: No bootstrap required, opened a new log
I20260927 22:50:05.815404 13749 ts_tablet_manager.cc:1397] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20260927 22:50:05.825584 13749 raft_consensus.cc:348] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.826090 13749 raft_consensus.cc:374] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:05.826195 13749 raft_consensus.cc:729] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49589ba99da1417a99454dbb9a74b456, State: Initialized, Role: FOLLOWER
I20260927 22:50:05.826922 13749 consensus_queue.cc:260] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.828267 13724 heartbeater.cc:503] Master 127.0.97.62:41917 was elected leader, sending a full tablet report...
I20260927 22:50:05.828745 13749 ts_tablet_manager.cc:1428] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260927 22:50:05.880051 13591 tablet.cc:2369] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:05.980470 13754 raft_consensus.cc:482] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:50:05.980753 13754 raft_consensus.cc:504] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.982545 13754 leader_election.cc:288] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f831a47c369a46babeaa1b65614bd44f (127.0.97.2:34179), b6945c336cb641669efe129b6ec2a784 (127.0.97.1:33487)
I20260927 22:50:05.987949 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.982903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59047 (local address 127.0.97.2:34179)
0927 22:50:05.983091 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.983097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.983347 (+ 250us) server_negotiation.cc:407] Connection header received
0927 22:50:05.983530 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.983535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.983606 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.983693 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:05.984059 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.984781 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.986358 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.987119 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.987226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.987324 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.987594 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.987710 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:50:05.987957 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.982901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34179 (local address 127.0.97.3:59047)
0927 22:50:05.983212 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:05.983231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:05.983373 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:05.983731 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:05.983735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:05.983761 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:05.983931 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.983940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.984918 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.984923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.986199 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.986210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.987259 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.987265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.987450 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.987485 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:50:05.987566 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 22:50:05.987696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260927 22:50:05.988250 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.983085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53853 (local address 127.0.97.1:33487)
0927 22:50:05.983281 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:05.983286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:05.983592 (+ 306us) server_negotiation.cc:407] Connection header received
0927 22:50:05.983726 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:05.983731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:05.983788 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:05.983856 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:05.984341 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.985048 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.986666 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.987426 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.987564 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.987658 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:05.987900 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:50:05.988018 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:50:05.988458 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:05.982940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33487 (local address 127.0.97.3:53853)
0927 22:50:05.983496 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:05.983511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:05.983610 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:05.983894 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:05.983898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:05.983920 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:05.984199 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.984207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.985164 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.985170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.986527 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:05.986538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:05.987623 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:05.987629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:05.987761 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:05.987783 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:50:05.987856 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:50:05.987973 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":26,"thread_start_us":233,"threads_started":1}
I20260927 22:50:05.988931 13545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d6e1a93fb416a8c1c5050d8df2f73" candidate_uuid: "49589ba99da1417a99454dbb9a74b456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f831a47c369a46babeaa1b65614bd44f" is_pre_election: true
I20260927 22:50:05.989372 13545 raft_consensus.cc:2454] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49589ba99da1417a99454dbb9a74b456 in term 0.
I20260927 22:50:05.989336 13401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d6e1a93fb416a8c1c5050d8df2f73" candidate_uuid: "49589ba99da1417a99454dbb9a74b456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6945c336cb641669efe129b6ec2a784" is_pre_election: true
I20260927 22:50:05.989820 13401 raft_consensus.cc:2454] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49589ba99da1417a99454dbb9a74b456 in term 0.
I20260927 22:50:05.989987 13612 leader_election.cc:302] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49589ba99da1417a99454dbb9a74b456, f831a47c369a46babeaa1b65614bd44f; no voters:
I20260927 22:50:05.990315 13754 raft_consensus.cc:2781] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:50:05.990442 13754 raft_consensus.cc:482] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:50:05.990541 13754 raft_consensus.cc:3037] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:05.992637 13754 raft_consensus.cc:504] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:05.993176 13754 leader_election.cc:288] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [CANDIDATE]: Term 1 election: Requested vote from peers f831a47c369a46babeaa1b65614bd44f (127.0.97.2:34179), b6945c336cb641669efe129b6ec2a784 (127.0.97.1:33487)
I20260927 22:50:05.993525 13545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d6e1a93fb416a8c1c5050d8df2f73" candidate_uuid: "49589ba99da1417a99454dbb9a74b456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f831a47c369a46babeaa1b65614bd44f"
I20260927 22:50:05.993700 13545 raft_consensus.cc:3037] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:05.993726 13401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d6e1a93fb416a8c1c5050d8df2f73" candidate_uuid: "49589ba99da1417a99454dbb9a74b456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6945c336cb641669efe129b6ec2a784"
I20260927 22:50:05.993928 13401 raft_consensus.cc:3037] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:05.995853 13545 raft_consensus.cc:2454] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49589ba99da1417a99454dbb9a74b456 in term 1.
I20260927 22:50:05.996088 13401 raft_consensus.cc:2454] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49589ba99da1417a99454dbb9a74b456 in term 1.
I20260927 22:50:05.996150 13612 leader_election.cc:302] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49589ba99da1417a99454dbb9a74b456, f831a47c369a46babeaa1b65614bd44f; no voters:
I20260927 22:50:05.996369 13754 raft_consensus.cc:2781] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:05.996870 13754 raft_consensus.cc:686] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 1 LEADER]: Becoming Leader. State: Replica: 49589ba99da1417a99454dbb9a74b456, State: Running, Role: LEADER
I20260927 22:50:05.997365 13754 consensus_queue.cc:237] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } }
I20260927 22:50:06.001663 13248 catalog_manager.cc:5495] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49589ba99da1417a99454dbb9a74b456 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "49589ba99da1417a99454dbb9a74b456" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49589ba99da1417a99454dbb9a74b456" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36773 } health_report { overall_health: HEALTHY } } }
W20260927 22:50:06.043848 13725 tablet.cc:2369] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:06.084089 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:06.078848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.0.97.3:36773)
0927 22:50:06.079028 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:06.079032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:06.079049 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:50:06.079196 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:06.079200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:06.079285 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:06.079488 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:50:06.079925 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.080674 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.081987 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.082377 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.082468 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:06.082970 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:50:06.083535 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:50:06.083652 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:06.083812 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 22:50:06.083881 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:50:06.091756 13545 raft_consensus.cc:1261] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 1 FOLLOWER]: Refusing update from remote peer 49589ba99da1417a99454dbb9a74b456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:06.091753 13401 raft_consensus.cc:1261] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 1 FOLLOWER]: Refusing update from remote peer 49589ba99da1417a99454dbb9a74b456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:06.092644 13763 consensus_queue.cc:1035] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6945c336cb641669efe129b6ec2a784" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:06.092975 13754 consensus_queue.cc:1035] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f831a47c369a46babeaa1b65614bd44f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:06.103390 13766 mvcc.cc:205] Tried to move back new op lower bound from 7334090367342678016 to 7334090366968680448. Current Snapshot: MvccSnapshot[applied={T|T < 7334090367342678016}]
I20260927 22:50:06.115269 13247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:06.115571 13247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:06.118546 13247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b550c4ed6f444366a6f0ec7420c567dd: Sending DeleteTablet for 3 replicas of tablet d77d6e1a93fb416a8c1c5050d8df2f73
I20260927 22:50:06.119601 13525 tablet_service.cc:1514] Processing DeleteTablet for tablet d77d6e1a93fb416a8c1c5050d8df2f73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58652
I20260927 22:50:06.119642 13381 tablet_service.cc:1514] Processing DeleteTablet for tablet d77d6e1a93fb416a8c1c5050d8df2f73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35786
I20260927 22:50:06.119867 13659 tablet_service.cc:1514] Processing DeleteTablet for tablet d77d6e1a93fb416a8c1c5050d8df2f73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33114
I20260927 22:50:06.130612 13775 tablet_replica.cc:330] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: stopping tablet replica
I20260927 22:50:06.131199 13775 raft_consensus.cc:2229] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:06.131763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13285
I20260927 22:50:06.131927 13775 raft_consensus.cc:2258] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:06.132166 13776 tablet_replica.cc:330] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: stopping tablet replica
I20260927 22:50:06.132606 13776 raft_consensus.cc:2229] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:06.133095 13776 raft_consensus.cc:2258] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:06.133121 13777 tablet_replica.cc:330] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: stopping tablet replica
I20260927 22:50:06.133620 13777 raft_consensus.cc:2229] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:50:06.134249 13777 raft_consensus.cc:2258] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:06.136022 13775 ts_tablet_manager.cc:1905] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:06.136245 13777 ts_tablet_manager.cc:1905] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:06.136358 13776 ts_tablet_manager.cc:1905] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:06.142717 13775 ts_tablet_manager.cc:1918] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:06.142717 13777 ts_tablet_manager.cc:1918] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:06.142879 13775 log.cc:1196] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/d77d6e1a93fb416a8c1c5050d8df2f73
I20260927 22:50:06.142889 13777 log.cc:1196] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/d77d6e1a93fb416a8c1c5050d8df2f73
I20260927 22:50:06.143381 13775 ts_tablet_manager.cc:1939] T d77d6e1a93fb416a8c1c5050d8df2f73 P b6945c336cb641669efe129b6ec2a784: Deleting consensus metadata
I20260927 22:50:06.143370 13777 ts_tablet_manager.cc:1939] T d77d6e1a93fb416a8c1c5050d8df2f73 P 49589ba99da1417a99454dbb9a74b456: Deleting consensus metadata
I20260927 22:50:06.143388 13776 ts_tablet_manager.cc:1918] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:06.143548 13776 log.cc:1196] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/d77d6e1a93fb416a8c1c5050d8df2f73
I20260927 22:50:06.144062 13776 ts_tablet_manager.cc:1939] T d77d6e1a93fb416a8c1c5050d8df2f73 P f831a47c369a46babeaa1b65614bd44f: Deleting consensus metadata
I20260927 22:50:06.145273 13234 catalog_manager.cc:4841] TS b6945c336cb641669efe129b6ec2a784 (127.0.97.1:33487): tablet d77d6e1a93fb416a8c1c5050d8df2f73 (table test-table [id=4dbbe683cb914742a2876a92d841ef48]) successfully deleted
I20260927 22:50:06.146770 13233 catalog_manager.cc:4841] TS 49589ba99da1417a99454dbb9a74b456 (127.0.97.3:36773): tablet d77d6e1a93fb416a8c1c5050d8df2f73 (table test-table [id=4dbbe683cb914742a2876a92d841ef48]) successfully deleted
I20260927 22:50:06.147611 13232 catalog_manager.cc:4841] TS f831a47c369a46babeaa1b65614bd44f (127.0.97.2:34179): tablet d77d6e1a93fb416a8c1c5050d8df2f73 (table test-table [id=4dbbe683cb914742a2876a92d841ef48]) successfully deleted
I20260927 22:50:06.158782 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:06.132866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.0.97.1:33487)
0927 22:50:06.133059 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:06.133064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:06.146068 (+ 13004us) server_negotiation.cc:407] Connection header received
0927 22:50:06.146231 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:06.146236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:06.146317 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:06.146440 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:06.147897 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.148610 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.149814 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.150194 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.150298 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:06.152802 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:06.152813 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:06.152829 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:06.152862 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:06.155936 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:06.156636 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:06.156644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:06.156646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:06.156790 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:06.157171 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:06.157175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:06.157177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:06.157563 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:06.157878 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:06.158182 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:50:06.158392 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:50:06.159548 13441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:06.246726 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13285
I20260927 22:50:06.265911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13450
I20260927 22:50:06.280752 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:06.266871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.0.97.2:34179)
0927 22:50:06.267050 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:06.267056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:06.267075 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:50:06.267131 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:06.267135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:06.267208 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:06.267339 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:06.268954 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.269650 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.271346 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.271835 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.271924 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:06.274995 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:06.275008 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:06.275042 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:06.275077 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:06.277834 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:06.278576 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:06.278583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:06.278585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:06.278700 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:06.279048 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:06.279050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:06.279052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:06.279394 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:06.279657 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:06.280001 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 22:50:06.280211 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:50:06.281431 13585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:06.368819 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13450
I20260927 22:50:06.387988 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13594
I20260927 22:50:06.401257 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:06.388879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.0.97.3:36773)
0927 22:50:06.389063 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:06.389068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:06.389086 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:50:06.389161 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:06.389166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:06.389236 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:06.389328 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:06.390593 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.391422 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.392720 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:06.393166 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:06.393274 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:06.395724 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:06.395735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:06.395751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:06.395791 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:06.398648 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:06.399270 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:06.399281 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:06.399283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:06.399457 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:06.399792 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:06.399798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:06.399801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:06.400138 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:06.400452 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:06.400713 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:50:06.400905 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:50:06.402102 13719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:06.519881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13594
I20260927 22:50:06.535279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13215
I20260927 22:50:06.536521 13278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:06.647493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13215
I20260927 22:50:06.671259 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:50:07.005921 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:50:07.006131 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:50:07.341696 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:50:07.341 UTC [12268] LOG: received smart shutdown request
2026-09-27 22:50:07.344 UTC [12268] LOG: background worker "logical replication launcher" (PID 12277) exited with exit code 1
2026-09-27 22:50:07.344 UTC [12272] LOG: shutting down
2026-09-27 22:50:07.496 UTC [12268] LOG: database system is shut down
2026-09-27T22:50:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50619 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:50:07 dist-test-slave-kr25 krb5kdc[13791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:50:07 dist-test-slave-kr25 krb5kdc[13791](info): set up 2 sockets
Sep 27 22:50:07 dist-test-slave-kr25 krb5kdc[13791](info): commencing operation
krb5kdc: starting...
W20260927 22:50:08.611546 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:50:08 dist-test-slave-kr25 krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:50:08.826208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34221
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34221,127.0.97.61:40197,127.0.97.60:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:09.141969 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:09.142292 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:09.142424 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:09.149963 13804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:09.150076 13804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:09.150136 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:09.150230 13804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:09.150288 13804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:09.160377 13804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34221,127.0.97.61:40197,127.0.97.60:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:09.161836 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:09.163087 13804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:09.174567 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:09.174485 13812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:09.174449 13810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:09.176002 13804 server_base.cc:1034] running on GCE node
I20260927 22:50:09.176843 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:09.178426 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:09.181806 13804 hybrid_clock.cc:648] HybridClock initialized: now 1790549409181739 us; error 64 us; skew 500 ppm
I20260927 22:50:09.182204 13804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:09 dist-test-slave-kr25 krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:09.185413 13804 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:50:09.186864 13804 webserver.cc:460] Webserver started at http://127.0.97.62:45663/ using document root <none> and password file <none>
I20260927 22:50:09.187620 13804 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:09.187731 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:09.188035 13804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:09.191007 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "8ee02cd4277348dfa0a5dd691f02656e"
format_stamp: "Formatted at 2026-09-27 22:50:09 on dist-test-slave-kr25"
server_key: "ab7b5efaf0213831ae53f1492ebdf4bc"
server_key_iv: "3076a49689f3612fdb770cd0af5628a7"
server_key_version: "encryptionkey@0"
I20260927 22:50:09.191768 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ee02cd4277348dfa0a5dd691f02656e"
format_stamp: "Formatted at 2026-09-27 22:50:09 on dist-test-slave-kr25"
server_key: "ab7b5efaf0213831ae53f1492ebdf4bc"
server_key_iv: "3076a49689f3612fdb770cd0af5628a7"
server_key_version: "encryptionkey@0"
I20260927 22:50:09.196686 13804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:50:09.200024 13819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:09.201406 13804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:50:09.201555 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "8ee02cd4277348dfa0a5dd691f02656e"
format_stamp: "Formatted at 2026-09-27 22:50:09 on dist-test-slave-kr25"
server_key: "ab7b5efaf0213831ae53f1492ebdf4bc"
server_key_iv: "3076a49689f3612fdb770cd0af5628a7"
server_key_version: "encryptionkey@0"
I20260927 22:50:09.201740 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:09.218238 13804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:09.222538 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:09.222796 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:09.240370 13804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34221
I20260927 22:50:09.240445 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34221 every 8 connection(s)
I20260927 22:50:09.241752 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:50:09.245229 13871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:09.246536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13804
I20260927 22:50:09.246773 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:50:09.247561 388 external_mini_cluster.cc:1428] Setting key 815174d0da0b121b8479db630497de96
I20260927 22:50:09.253631 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 22:50:09 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:09.278594 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.257305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34221 (local address 127.0.0.1:56842)
0927 22:50:09.258531 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.258572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:09.260339 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:09.261255 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:09.261260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:09.261792 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:09.262128 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.262145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.264257 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.264261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.265401 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.265413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.266609 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.266614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.266720 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.267318 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:09.267339 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:09.271830 (+ 4491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:09.274638 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.274648 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.274659 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.275030 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.275385 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.275390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.275394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.275591 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.276315 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:09.276324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:09.276551 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 22:50:09.276765 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:50:09.277313 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":116,"spinlock_wait_cycles":48000,"thread_start_us":743,"threads_started":1}
I20260927 22:50:09.278611 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.257303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.0.97.62:34221)
0927 22:50:09.258143 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:09.258157 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:09.258714 (+ 557us) server_negotiation.cc:407] Connection header received
0927 22:50:09.260704 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:09.260712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:09.261024 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:09.261231 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:09.262313 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.264111 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.265543 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.266387 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.266598 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.272040 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:09.272048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:09.272059 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:09.272087 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:09.274526 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.275140 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.275145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.275148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.275286 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.275683 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.275688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.275691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.276183 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:09.276377 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:09.276904 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 22:50:09.277177 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":373,"threads_started":1}
I20260927 22:50:09.281581 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:50:09.282531 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.282171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32920)
0927 22:50:09.282323 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.282418 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:50:09.282774 13823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
I20260927 22:50:09.284260 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.283945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32926)
0927 22:50:09.284070 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.284135 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:50:09.284452 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40197: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:50:09.296271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40197
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34221,127.0.97.61:40197,127.0.97.60:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:50:09.333772 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } attempt: 1
I20260927 22:50:09.334918 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.334462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32930)
0927 22:50:09.334658 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.334761 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:50:09.336081 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.335721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32932)
0927 22:50:09.335870 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.335946 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:50:09.336306 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40197: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
I20260927 22:50:09.404556 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } attempt: 2
I20260927 22:50:09.405766 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.405307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32946)
0927 22:50:09.405547 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.405640 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:50:09.406761 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.406425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32948)
0927 22:50:09.406568 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.406634 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:50:09.406942 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40197: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
I20260927 22:50:09.498198 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } attempt: 3
I20260927 22:50:09.499348 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.498911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32954)
0927 22:50:09.499111 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.499199 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:50:09.500480 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.500090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32956)
0927 22:50:09.500237 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.500349 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:50:09.500684 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40197: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
W20260927 22:50:09.609759 13878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:09.610042 13878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:09.610128 13878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:09.617236 13878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:09.617359 13878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:09.617413 13878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:09.617468 13878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:09.617511 13878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:09.627233 13878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34221,127.0.97.61:40197,127.0.97.60:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:09.628535 13878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:09.629755 13878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:09.640658 13883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:09.640743 13886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:09.640713 13884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:09.641508 13878 server_base.cc:1034] running on GCE node
I20260927 22:50:09.642194 13878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:09.643520 13878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:09.643882 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } attempt: 4
I20260927 22:50:09.644726 13878 hybrid_clock.cc:648] HybridClock initialized: now 1790549409644662 us; error 37 us; skew 500 ppm
I20260927 22:50:09.644927 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.644398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32968)
0927 22:50:09.644653 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.644761 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260927 22:50:09.645136 13878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:50:09.645912 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.645608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32972)
0927 22:50:09.645728 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.645805 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 22:50:09.646402 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40197: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40197: connect: Connection refused (error 111)
Sep 27 22:50:09 dist-test-slave-kr25 krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:09.648245 13878 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:50:09.649641 13878 webserver.cc:460] Webserver started at http://127.0.97.61:37729/ using document root <none> and password file <none>
I20260927 22:50:09.650286 13878 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:09.650398 13878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:09.650668 13878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:09.653568 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data/instance:
uuid: "adb8db9a723f43fb9238a9af6f23ba33"
format_stamp: "Formatted at 2026-09-27 22:50:09 on dist-test-slave-kr25"
server_key: "ddcf99728263b8d647125bbf6870f5a0"
server_key_iv: "22439e671dec2e7dc0d767878d267d52"
server_key_version: "encryptionkey@0"
I20260927 22:50:09.654335 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/wal/instance:
uuid: "adb8db9a723f43fb9238a9af6f23ba33"
format_stamp: "Formatted at 2026-09-27 22:50:09 on dist-test-slave-kr25"
server_key: "ddcf99728263b8d647125bbf6870f5a0"
server_key_iv: "22439e671dec2e7dc0d767878d267d52"
server_key_version: "encryptionkey@0"
I20260927 22:50:09.659538 13878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:50:09.662837 13893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:09.664300 13878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:50:09.664455 13878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/wal
uuid: "adb8db9a723f43fb9238a9af6f23ba33"
format_stamp: "Formatted at 2026-09-27 22:50:09 on dist-test-slave-kr25"
server_key: "ddcf99728263b8d647125bbf6870f5a0"
server_key_iv: "22439e671dec2e7dc0d767878d267d52"
server_key_version: "encryptionkey@0"
I20260927 22:50:09.664629 13878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:09.685132 13878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:09.689006 13878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:09.689239 13878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:09.706588 13878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40197
I20260927 22:50:09.706630 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40197 every 8 connection(s)
I20260927 22:50:09.707947 13878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:50:09.711516 13945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:09.716522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13878
I20260927 22:50:09.716682 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-1/wal/instance
I20260927 22:50:09.717083 388 external_mini_cluster.cc:1428] Setting key f7e5b358a84992fc6d387195425adf8a
I20260927 22:50:09.717765 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:50:09 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 22:50:09.738291 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.720251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.0.97.62:34221)
0927 22:50:09.720471 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:09.720478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:09.721603 (+ 1125us) server_negotiation.cc:407] Connection header received
0927 22:50:09.723859 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:09.723866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:09.723956 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:09.724061 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:09.725490 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.726445 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.727959 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.728572 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.728692 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.732099 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:09.732106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:09.732111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:09.732153 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:09.735378 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.736211 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.736216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.736218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.736339 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.736674 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.736677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.736679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.737010 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:09.737122 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:09.737783 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 22:50:09.738033 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 22:50:09.739382 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.720684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34221 (local address 127.0.0.1:56846)
0927 22:50:09.721423 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.721473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:09.723592 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:09.724269 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:09.724277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:09.724757 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:09.725333 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.725351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.726577 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.726581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.727821 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.727833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.728753 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.728757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.728958 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.729579 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:09.729601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:09.731875 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:09.735576 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.735582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.735597 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.736078 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.736421 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.736424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.736426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.736590 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.737109 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:09.737115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:09.737457 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 22:50:09.737690 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 22:50:09.738080 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":291,"threads_started":1}
I20260927 22:50:09.741305 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:50:09 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:50:09.759956 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.741922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32982)
0927 22:50:09.742792 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.742811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:09.742915 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:09.743374 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:09.743379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:09.743798 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:09.743992 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.744001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.745593 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.745598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.746722 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.746734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.747773 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.747777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.747980 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.751718 (+ 3738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:09.751746 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:09.754145 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:09.757459 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.757463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.757466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.757805 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.758179 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.758182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.758184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.758320 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.758983 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:09.758987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:09.759179 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 22:50:09.759404 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 22:50:09.759658 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":28}
I20260927 22:50:09.760089 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.742172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.0.97.61:40197)
0927 22:50:09.742823 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:09.742831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:09.742869 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:50:09.743018 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:09.743025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:09.743238 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:09.743360 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:09.744136 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.745445 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.746842 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.747655 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.747764 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.754416 (+ 6652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:09.754431 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:09.754447 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:09.754484 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:09.757326 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.757937 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.757948 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.757951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.758085 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.758410 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.758413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.758415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.758841 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:09.759004 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:09.759582 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 22:50:09.759765 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":486,"thread_start_us":358,"threads_started":1}
I20260927 22:50:09.762298 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:50:09.763352 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.762954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52530)
0927 22:50:09.763110 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.763205 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:50:09.763633 13896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
I20260927 22:50:09.765045 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.764754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52534)
0927 22:50:09.764889 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.764959 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 22:50:09.765246 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:50:09.766644 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35191
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34221,127.0.97.61:40197,127.0.97.60:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:50:09.814556 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 1
I20260927 22:50:09.815675 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.815176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52546)
0927 22:50:09.815355 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.815521 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:50:09.816588 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.816270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52562)
0927 22:50:09.816424 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.816487 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 22:50:09.816780 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
I20260927 22:50:09.884994 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 2
I20260927 22:50:09.886101 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.885674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52574)
0927 22:50:09.885867 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.885954 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:50:09.887013 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.886695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52582)
0927 22:50:09.886834 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.886902 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:50:09.887219 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
I20260927 22:50:09.945683 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } attempt: 5
Sep 27 22:50:09 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:50:09.958349 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.946456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:32992)
0927 22:50:09.946645 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.946666 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:09.946795 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:09.947148 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:09.947154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:09.947600 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:09.947778 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.947788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.948757 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.948761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.949621 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:09.949634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.950368 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.950374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:09.950551 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.951106 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:09.951123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:09.953300 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:09.956160 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.956166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.956170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.956520 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.956841 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:09.956846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:09.956850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:09.956987 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:09.957469 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:09.957474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:09.957613 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 22:50:09.957726 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:50:09.958005 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:50:09.958349 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.946592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.0.97.61:40197)
0927 22:50:09.946778 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:09.946784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:09.946804 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:50:09.946944 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:09.946950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:09.947011 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:09.947089 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:09.947931 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.948609 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.949765 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:09.950213 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:09.950332 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:09.953485 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:09.953490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:09.953493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:09.953523 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:09.956028 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.956624 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.956629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.956631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.956741 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:09.957082 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:09.957087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:09.957090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:09.957354 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:09.957486 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:09.957840 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 22:50:09.958037 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:50:09.959471 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:50:09.960557 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.960196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52594)
0927 22:50:09.960354 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.960430 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:50:09.961529 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.961219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52598)
0927 22:50:09.961365 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.961438 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:50:09.961735 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
I20260927 22:50:09.978433 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 3
I20260927 22:50:09.979516 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.979054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52602)
0927 22:50:09.979236 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.979324 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 22:50:09.980664 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:09.980107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52616)
0927 22:50:09.980471 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:09.980548 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272}
W20260927 22:50:09.980849 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
I20260927 22:50:10.012951 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 1
I20260927 22:50:10.014148 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.013672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52618)
0927 22:50:10.013885 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.013987 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:50:10.015137 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.014814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52622)
0927 22:50:10.014948 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.015031 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:50:10.015415 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
W20260927 22:50:10.079897 13952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:10.080179 13952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:10.080274 13952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:50:10.083674 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 2
I20260927 22:50:10.084812 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.084324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52632)
0927 22:50:10.084532 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.084630 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:50:10.085779 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.085474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52640)
0927 22:50:10.085602 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.085669 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:50:10.085968 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
W20260927 22:50:10.087900 13952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:10.087997 13952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:10.088063 13952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:10.088137 13952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:10.088193 13952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:10.098407 13952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34221,127.0.97.61:40197,127.0.97.60:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:10.099833 13952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:10.101133 13952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:10.112342 13957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:10.112294 13958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:10.112387 13960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:10.113042 13952 server_base.cc:1034] running on GCE node
I20260927 22:50:10.113732 13952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:10.115006 13952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:10.116245 13952 hybrid_clock.cc:648] HybridClock initialized: now 1790549410116202 us; error 33 us; skew 500 ppm
I20260927 22:50:10.116587 13952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:10.119788 13952 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:50:10.121358 13952 webserver.cc:460] Webserver started at http://127.0.97.60:45185/ using document root <none> and password file <none>
I20260927 22:50:10.122026 13952 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:10.122139 13952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:10.122442 13952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:10.124074 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 4
I20260927 22:50:10.125172 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.124597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52656)
0927 22:50:10.124860 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.124967 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:50:10.125838 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data/instance:
uuid: "cab1a5bc377a46a69ebf0053a94de62a"
format_stamp: "Formatted at 2026-09-27 22:50:10 on dist-test-slave-kr25"
server_key: "83278ef490ca6eefed9ee509ec4f2504"
server_key_iv: "5c06e05d2566c2e09bd5179a8e05c0aa"
server_key_version: "encryptionkey@0"
I20260927 22:50:10.126286 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.125945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52658)
0927 22:50:10.126109 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.126179 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 22:50:10.126576 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35191: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35191: connect: Connection refused (error 111)
I20260927 22:50:10.126842 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/wal/instance:
uuid: "cab1a5bc377a46a69ebf0053a94de62a"
format_stamp: "Formatted at 2026-09-27 22:50:10 on dist-test-slave-kr25"
server_key: "83278ef490ca6eefed9ee509ec4f2504"
server_key_iv: "5c06e05d2566c2e09bd5179a8e05c0aa"
server_key_version: "encryptionkey@0"
I20260927 22:50:10.131800 13952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:50:10.135233 13967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:10.136781 13952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:50:10.136953 13952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/wal
uuid: "cab1a5bc377a46a69ebf0053a94de62a"
format_stamp: "Formatted at 2026-09-27 22:50:10 on dist-test-slave-kr25"
server_key: "83278ef490ca6eefed9ee509ec4f2504"
server_key_iv: "5c06e05d2566c2e09bd5179a8e05c0aa"
server_key_version: "encryptionkey@0"
I20260927 22:50:10.137152 13952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:10.162215 13952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:10.166144 13952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:10.166370 13952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:10.183660 13952 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35191
I20260927 22:50:10.183691 14018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35191 every 8 connection(s)
I20260927 22:50:10.184975 13952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:50:10.186537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 13952
I20260927 22:50:10.186700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549216186161-388-0/minicluster-data/master-2/wal/instance
I20260927 22:50:10.187094 388 external_mini_cluster.cc:1428] Setting key a90da4debae044c5c7b4cf23c6650f2e
I20260927 22:50:10.188845 14019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:10.192139 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 3
I20260927 22:50:10.199148 14019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:50:10.209434 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.192681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52676)
0927 22:50:10.192884 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.192902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:10.193015 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:10.193899 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:10.193904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:10.194339 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:10.194508 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.194517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.196337 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.196342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.197229 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.197240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.198274 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.198280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.198471 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.199203 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:10.199226 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:10.201895 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:10.206182 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.206187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.206190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.206643 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.207126 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.207131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.207135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.207287 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.208203 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:10.208209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:10.208565 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 22:50:10.208729 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:50:10.209001 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":152}
I20260927 22:50:10.210764 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.193053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.0.97.60:35191)
0927 22:50:10.193600 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.193605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.193626 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:50:10.193668 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.193672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.193729 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.193850 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.194665 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.196190 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.197355 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.198074 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.198275 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.202199 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.202209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.202228 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.202284 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.206050 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.206766 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.206774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.206782 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.207007 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.207457 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.207461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.207464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.208069 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.208386 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.209830 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.210022 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":435,"thread_start_us":310,"threads_started":1}
I20260927 22:50:10.210801 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.188846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.0.97.60:35191)
0927 22:50:10.189620 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.189634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.189683 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:50:10.191656 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.191683 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.192183 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.192600 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.194439 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.196241 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.198124 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.198643 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.198750 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.202997 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.203003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.203007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.203036 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.206877 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.207641 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.207645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.207648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.207777 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.208142 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.208146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.208150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.208489 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.208691 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.209179 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.209623 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":557,"thread_start_us":305,"threads_started":1}
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:10.217886 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e: Bootstrap starting.
I20260927 22:50:10.218186 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.201972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34221 (local address 127.0.0.1:56858)
0927 22:50:10.202806 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.202848 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:10.203040 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:10.203396 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:10.203407 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:10.204281 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:10.204730 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.204743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.206062 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.206067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.207371 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.207387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.208257 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.208264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.208467 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.209056 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:10.209078 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:10.211695 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:10.215244 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.215255 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.215263 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.215772 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.216239 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.216244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.216248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.216426 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.217078 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:10.217087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:10.217262 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 22:50:10.217433 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 22:50:10.217763 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":3,"thread_start_us":418,"threads_started":1}
I20260927 22:50:10.218289 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.201921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.0.97.62:34221)
0927 22:50:10.202103 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.202108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.202928 (+ 820us) server_negotiation.cc:407] Connection header received
0927 22:50:10.203162 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.203168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.203244 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.203336 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.204868 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.205891 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.207530 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.208096 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.208204 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.211880 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.211887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.211892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.211929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.215126 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.215931 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.215936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.215940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.216107 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.216567 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.216572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.216575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.216933 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.217119 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.217587 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.217796 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":64}
I20260927 22:50:10.219615 14019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:50:10.223239 13871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:10.224584 13871 log.cc:824] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:50:10.232007 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.214408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.0.97.62:34221)
0927 22:50:10.215115 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.215121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.215147 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:50:10.215192 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.215197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.215274 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.215456 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.217038 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.218320 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.220066 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.220611 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.220788 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.225884 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.225891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.225896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.225934 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.229116 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.229945 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.229950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.229954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.230116 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.230455 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.230460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.230464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.230760 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.231044 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.231355 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.231594 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":371,"threads_started":1}
I20260927 22:50:10.232681 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.220242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40197 (local address 127.0.0.1:33004)
0927 22:50:10.220420 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.220465 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:10.220573 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:10.221083 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:10.221087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:10.221440 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:10.221612 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.221623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.222533 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.222538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.223386 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.223399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.224100 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.224105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.224287 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.224774 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:10.224795 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:10.226934 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:10.230115 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.230120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.230123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.230640 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.231035 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.231039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.231043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.231182 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.231841 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:10.231845 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:10.231993 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 22:50:10.232131 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:50:10.232394 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 22:50:10.233441 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.220361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.0.97.61:40197)
0927 22:50:10.220538 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.220543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.220561 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:50:10.220716 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.220723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.220806 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.220915 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.221741 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.222405 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.223535 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.223956 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.224061 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.227122 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.227128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.227133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.227173 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.229994 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.230758 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.230762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.230766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.230910 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.231302 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.231306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.231310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.231649 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.231784 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.232950 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.233116 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:50:10.233892 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e: No bootstrap required, opened a new log
I20260927 22:50:10.234253 14019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:50:10.245729 13871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.246340 13871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:10.246450 13871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee02cd4277348dfa0a5dd691f02656e, State: Initialized, Role: FOLLOWER
I20260927 22:50:10.247164 13871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.248301 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.234920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52688)
0927 22:50:10.235298 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.235315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:10.235425 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:10.235729 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:10.235734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:10.236142 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:10.236312 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.236322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.237445 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.237450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.238292 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.238303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.239099 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.239105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.239290 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.239909 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:10.239931 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:10.242373 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:10.245798 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.245803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.245807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.246233 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.246612 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.246616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.246620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.246758 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.247338 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:10.247343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:10.247541 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 22:50:10.247691 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:50:10.247966 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260927 22:50:10.249085 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.234967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.0.97.60:35191)
0927 22:50:10.235147 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.235152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.235393 (+ 241us) server_negotiation.cc:407] Connection header received
0927 22:50:10.235542 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.235546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.235609 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.235700 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.236499 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.237339 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.238405 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.238959 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.239073 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.242553 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.242559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.242562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.242596 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.245666 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.246346 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.246351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.246354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.246502 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.246862 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.246865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.246868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.247165 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.247293 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.248572 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.248777 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:50:10.250931 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.233810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.0.97.61:40197)
0927 22:50:10.233952 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.233958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.234214 (+ 256us) server_negotiation.cc:407] Connection header received
0927 22:50:10.234345 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.234350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.234424 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.234504 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.235885 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.236881 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.238591 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.239190 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.239289 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.244244 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.244250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.244254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.244288 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.247732 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.248691 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.248696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.248699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.248837 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.249460 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.249464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.249467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.249780 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.250015 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.250439 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.250635 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:50:10.254917 14019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: Bootstrap starting.
I20260927 22:50:10.255136 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:50:10.259260 14019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:10.260713 14019 log.cc:824] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:10.261797 14027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.262400 14027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:50:10.263113 13871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:10.265683 14019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: No bootstrap required, opened a new log
I20260927 22:50:10.275875 14019 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.276381 14019 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:10.276491 14019 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab1a5bc377a46a69ebf0053a94de62a, State: Initialized, Role: FOLLOWER
I20260927 22:50:10.277211 14019 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.287389 14047 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:50:10.290298 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.271825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.0.97.62:34221)
0927 22:50:10.272033 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.272039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.272809 (+ 770us) server_negotiation.cc:407] Connection header received
0927 22:50:10.273248 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.273254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.273328 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.273412 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.275641 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.276756 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.279109 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.279792 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.279888 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.283620 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.283626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.283631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.283670 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.287144 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.288057 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.288062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.288066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.288220 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.288640 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.288644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.288647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.288967 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.289195 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.289747 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.289958 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:50:10.291956 13804 master_runner.cc:186] Error getting master registration for 127.0.97.61:40197: OK, instance_id { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" instance_seqno: 1790549409701673 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:50:10.301576 13804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:50:10.304303 14046 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.304807 14046 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:50:10.305800 14019 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:10.307364 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.272727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.0.97.61:40197)
0927 22:50:10.272951 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.272956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.291203 (+ 18247us) server_negotiation.cc:407] Connection header received
0927 22:50:10.291468 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.291474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.291548 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.291615 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.293044 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.294102 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.296043 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.296694 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.296790 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.300091 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.300097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.300103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.300145 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.303799 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.305066 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.305072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.305076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.305212 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.305609 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.305614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.305618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.305999 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.306261 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.306786 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.307028 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 22:50:10.308820 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.269562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.0.97.60:35191)
0927 22:50:10.269754 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.269760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.271767 (+ 2007us) server_negotiation.cc:407] Connection header received
0927 22:50:10.272743 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.272749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.272831 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.272919 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.277561 (+ 4642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.278622 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.282835 (+ 4213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.283503 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.283587 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.302177 (+ 18590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.302185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.302190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.302234 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.305640 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.306516 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.306522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.306526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.306707 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.307143 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.307148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.307152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.307630 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.307838 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.308298 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.308497 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:50:10.321376 14058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:50:10.321775 14058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:50:10.322054 13952 master_runner.cc:186] Error getting master registration for 127.0.97.61:40197: OK, instance_id { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" instance_seqno: 1790549409701673 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:50:10.329275 13952 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 22:50:10.418443 13973 tablet.cc:2369] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:10.425769 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } attempt: 5
Sep 27 22:50:10 dist-test-slave-kr25 krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:50:10.440552 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.426332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35191 (local address 127.0.0.1:52694)
0927 22:50:10.426571 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:10.426590 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:10.426706 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:10.427161 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:10.427167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:10.427713 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:10.427918 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.427931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.429058 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.429063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.430494 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:10.430510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.431273 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.431279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:10.431518 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.432138 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:10.432160 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:10.434607 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:10.438187 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.438190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.438193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.438562 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.438914 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:10.438918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:10.438920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:10.439044 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:10.439708 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:10.439712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:10.439842 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 22:50:10.439965 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:50:10.440178 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 22:50:10.440685 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:10.426576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.0.97.60:35191)
0927 22:50:10.426749 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:10.426755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:10.426774 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:50:10.426856 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:10.426861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:10.426966 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:10.427058 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:10.428096 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.428924 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.430636 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:10.431131 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:10.431237 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:10.434780 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:10.434785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:10.434788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:10.434828 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:10.438054 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.438666 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.438671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.438675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.438819 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:10.439135 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:10.439139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:10.439142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:10.439593 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:10.439738 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:10.440072 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 22:50:10.440324 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:50:10.445664 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: Bootstrap starting.
I20260927 22:50:10.449031 13945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:10.449988 13945 log.cc:824] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:10.452543 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: No bootstrap required, opened a new log
I20260927 22:50:10.460198 13945 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.460534 13945 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:10.460646 13945 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8db9a723f43fb9238a9af6f23ba33, State: Initialized, Role: FOLLOWER
I20260927 22:50:10.461253 13945 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.462124 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.462496 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:50:10.463549 13945 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:50:10.472801 14072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:50:10.472937 14072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 22:50:10.475591 13825 tablet.cc:2369] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:10.478324 13878 master_runner.cc:263] No leader master found from master adb8db9a723f43fb9238a9af6f23ba33
I20260927 22:50:10.478482 13878 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:50:10.642877 14027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:50:10.643187 14027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.644785 14027 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adb8db9a723f43fb9238a9af6f23ba33 (127.0.97.61:40197), cab1a5bc377a46a69ebf0053a94de62a (127.0.97.60:35191)
I20260927 22:50:10.645596 13994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ee02cd4277348dfa0a5dd691f02656e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab1a5bc377a46a69ebf0053a94de62a" is_pre_election: true
I20260927 22:50:10.645568 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ee02cd4277348dfa0a5dd691f02656e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8db9a723f43fb9238a9af6f23ba33" is_pre_election: true
I20260927 22:50:10.646129 13920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee02cd4277348dfa0a5dd691f02656e in term 0.
I20260927 22:50:10.646137 13994 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee02cd4277348dfa0a5dd691f02656e in term 0.
I20260927 22:50:10.646718 13823 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee02cd4277348dfa0a5dd691f02656e, adb8db9a723f43fb9238a9af6f23ba33; no voters:
I20260927 22:50:10.647174 14027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:50:10.647346 14027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:50:10.647475 14027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:10.650564 14027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.651629 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ee02cd4277348dfa0a5dd691f02656e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb8db9a723f43fb9238a9af6f23ba33"
I20260927 22:50:10.651693 13994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ee02cd4277348dfa0a5dd691f02656e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab1a5bc377a46a69ebf0053a94de62a"
I20260927 22:50:10.651852 13920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:10.651885 13994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:10.655107 13920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee02cd4277348dfa0a5dd691f02656e in term 1.
I20260927 22:50:10.655191 13994 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee02cd4277348dfa0a5dd691f02656e in term 1.
I20260927 22:50:10.655675 13822 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee02cd4277348dfa0a5dd691f02656e, cab1a5bc377a46a69ebf0053a94de62a; no voters:
I20260927 22:50:10.655771 14027 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [CANDIDATE]: Term 1 election: Requested vote from peers adb8db9a723f43fb9238a9af6f23ba33 (127.0.97.61:40197), cab1a5bc377a46a69ebf0053a94de62a (127.0.97.60:35191)
I20260927 22:50:10.655983 14027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:10.656214 14027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [term 1 LEADER]: Becoming Leader. State: Replica: 8ee02cd4277348dfa0a5dd691f02656e, State: Running, Role: LEADER
I20260927 22:50:10.658573 14027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } }
I20260927 22:50:10.673018 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee02cd4277348dfa0a5dd691f02656e. Latest consensus state: current_term: 1 leader_uuid: "8ee02cd4277348dfa0a5dd691f02656e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.673548 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:10.674597 14076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:50:10.679741 14076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 22:50:10.691679 13899 tablet.cc:2369] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:10.694527 13994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [term 1 FOLLOWER]: Refusing update from remote peer 8ee02cd4277348dfa0a5dd691f02656e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:10.694533 13920 raft_consensus.cc:1261] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [term 1 FOLLOWER]: Refusing update from remote peer 8ee02cd4277348dfa0a5dd691f02656e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:10.695683 14074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:10.696082 14027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:10.727957 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee02cd4277348dfa0a5dd691f02656e. Latest consensus state: current_term: 1 leader_uuid: "8ee02cd4277348dfa0a5dd691f02656e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.728293 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:50:10.730728 14078 mvcc.cc:205] Tried to move back new op lower bound from 7334090386192408576 to 7334090386071601152. Current Snapshot: MvccSnapshot[applied={T|T < 7334090386192408576}]
I20260927 22:50:10.732620 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ee02cd4277348dfa0a5dd691f02656e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.732893 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:10.734362 14076 catalog_manager.cc:1347] Generated new cluster ID: 6eccfa5e5f9d4ada9ab85316201b72b4
I20260927 22:50:10.734452 14076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:50:10.734338 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ee02cd4277348dfa0a5dd691f02656e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.734565 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:50:10.734731 14079 mvcc.cc:205] Tried to move back new op lower bound from 7334090386192408576 to 7334090386071601152. Current Snapshot: MvccSnapshot[applied={T|T < 7334090386192408576}]
I20260927 22:50:10.736838 14046 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee02cd4277348dfa0a5dd691f02656e. Latest consensus state: current_term: 1 leader_uuid: "8ee02cd4277348dfa0a5dd691f02656e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.737111 14046 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:50:10.741393 14027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ee02cd4277348dfa0a5dd691f02656e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.741668 14027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:10.759900 14046 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ee02cd4277348dfa0a5dd691f02656e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee02cd4277348dfa0a5dd691f02656e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34221 } } peers { permanent_uuid: "adb8db9a723f43fb9238a9af6f23ba33" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40197 } } peers { permanent_uuid: "cab1a5bc377a46a69ebf0053a94de62a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35191 } } }
I20260927 22:50:10.760166 14046 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:50:10.761263 14076 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:50:10.765164 14076 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:50:10.788527 14076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee02cd4277348dfa0a5dd691f02656e: Generated new TSK 0
I20260927 22:50:10.789835 14076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:50:11.323706 14058 catalog_manager.cc:1259] Loaded cluster ID: 6eccfa5e5f9d4ada9ab85316201b72b4
I20260927 22:50:11.323804 14058 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: loading cluster ID for follower catalog manager: success
I20260927 22:50:11.326092 14058 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: acquiring CA information for follower catalog manager: success
I20260927 22:50:11.327850 14058 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cab1a5bc377a46a69ebf0053a94de62a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:50:11.474772 14072 catalog_manager.cc:1259] Loaded cluster ID: 6eccfa5e5f9d4ada9ab85316201b72b4
I20260927 22:50:11.474881 14072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: loading cluster ID for follower catalog manager: success
I20260927 22:50:11.477339 14072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: acquiring CA information for follower catalog manager: success
I20260927 22:50:11.479039 14072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P adb8db9a723f43fb9238a9af6f23ba33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:50:11.875679 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13804
I20260927 22:50:11.876883 13866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:11.985255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13804
I20260927 22:50:12.002048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13878
I20260927 22:50:12.003202 13940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:12.107535 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13878
I20260927 22:50:12.123415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 13952
I20260927 22:50:12.124554 14014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:12.228880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 13952
2026-09-27T22:50:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4626 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:50:12 dist-test-slave-kr25 krb5kdc[14097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:50:12 dist-test-slave-kr25 krb5kdc[14097](info): set up 2 sockets
Sep 27 22:50:12 dist-test-slave-kr25 krb5kdc[14097](info): commencing operation
krb5kdc: starting...
W20260927 22:50:13.239645 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:50:13 dist-test-slave-kr25 krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:50:13.455077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42739
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:13.767222 14110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:13.767539 14110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:13.767644 14110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:13.774596 14110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:13.774698 14110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:13.774744 14110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:13.774797 14110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:13.774839 14110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:13.784647 14110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:13.785872 14110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:13.787107 14110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:13.798239 14115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:13.798213 14116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:13.798254 14118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:13.799118 14110 server_base.cc:1034] running on GCE node
I20260927 22:50:13.799891 14110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:13.801115 14110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:13.802311 14110 hybrid_clock.cc:648] HybridClock initialized: now 1790549413802269 us; error 27 us; skew 500 ppm
I20260927 22:50:13.802592 14110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:13 dist-test-slave-kr25 krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:13.805444 14110 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:50:13.806775 14110 webserver.cc:460] Webserver started at http://127.0.97.62:35121/ using document root <none> and password file <none>
I20260927 22:50:13.807377 14110 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:13.807515 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:13.807766 14110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:13.810467 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "5fdf094dd4f040c49630dd5ca87217f8"
format_stamp: "Formatted at 2026-09-27 22:50:13 on dist-test-slave-kr25"
server_key: "a115542bc30e18885c999b525b03d53f"
server_key_iv: "a6840a7cbef6a4b805758833cbcd9f5f"
server_key_version: "encryptionkey@0"
I20260927 22:50:13.811127 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "5fdf094dd4f040c49630dd5ca87217f8"
format_stamp: "Formatted at 2026-09-27 22:50:13 on dist-test-slave-kr25"
server_key: "a115542bc30e18885c999b525b03d53f"
server_key_iv: "a6840a7cbef6a4b805758833cbcd9f5f"
server_key_version: "encryptionkey@0"
I20260927 22:50:13.815954 14110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:50:13.819496 14125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:13.820909 14110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:50:13.821069 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "5fdf094dd4f040c49630dd5ca87217f8"
format_stamp: "Formatted at 2026-09-27 22:50:13 on dist-test-slave-kr25"
server_key: "a115542bc30e18885c999b525b03d53f"
server_key_iv: "a6840a7cbef6a4b805758833cbcd9f5f"
server_key_version: "encryptionkey@0"
I20260927 22:50:13.821251 14110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:13.836206 14110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:13.840205 14110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:13.840458 14110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:13.857390 14110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42739
I20260927 22:50:13.857438 14176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42739 every 8 connection(s)
I20260927 22:50:13.858767 14110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:50:13.862507 14177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:13.864988 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14110
I20260927 22:50:13.865186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:50:13.865944 388 external_mini_cluster.cc:1428] Setting key 8b3f7e01e92432a276b3b1787129ff15
I20260927 22:50:13.874498 14177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8: Bootstrap starting.
I20260927 22:50:13.878155 14177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:13.879384 14177 log.cc:824] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:50:13 dist-test-slave-kr25 krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:13.882408 14177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8: No bootstrap required, opened a new log
I20260927 22:50:13.888353 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:13.867858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.97.62:42739)
0927 22:50:13.868611 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:13.868622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:13.868664 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:50:13.870310 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:13.870332 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:13.870715 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:13.871057 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:13.872780 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:13.874050 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:13.875824 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:13.876506 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:13.876634 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:13.881362 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:13.881370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:13.881385 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:13.881418 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:13.884364 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:13.885161 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:13.885167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:13.885172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:13.885334 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:13.885691 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:13.885694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:13.885696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:13.886230 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:13.886441 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:13.886919 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 22:50:13.887318 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":341,"threads_started":1}
I20260927 22:50:13.890702 14177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdf094dd4f040c49630dd5ca87217f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42739 } }
I20260927 22:50:13.891122 14177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:13.891239 14177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdf094dd4f040c49630dd5ca87217f8, State: Initialized, Role: FOLLOWER
I20260927 22:50:13.891805 14177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdf094dd4f040c49630dd5ca87217f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42739 } }
I20260927 22:50:13.892007 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:50:13.892069 14177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:50:13.892187 14177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:50:13.892333 14177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:13.895255 14177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdf094dd4f040c49630dd5ca87217f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42739 } }
I20260927 22:50:13.895789 14177 leader_election.cc:302] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fdf094dd4f040c49630dd5ca87217f8; no voters:
I20260927 22:50:13.896318 14177 leader_election.cc:288] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:50:13.896426 14182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:13.897068 14182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [term 1 LEADER]: Becoming Leader. State: Replica: 5fdf094dd4f040c49630dd5ca87217f8, State: Running, Role: LEADER
I20260927 22:50:13.897588 14182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdf094dd4f040c49630dd5ca87217f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42739 } }
I20260927 22:50:13.898169 14177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:13.900223 14183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fdf094dd4f040c49630dd5ca87217f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdf094dd4f040c49630dd5ca87217f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42739 } } }
I20260927 22:50:13.900583 14183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:13.901578 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fdf094dd4f040c49630dd5ca87217f8. Latest consensus state: current_term: 1 leader_uuid: "5fdf094dd4f040c49630dd5ca87217f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdf094dd4f040c49630dd5ca87217f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42739 } } }
I20260927 22:50:13.901844 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:13.904198 14189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:50:13.909183 14189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:50:13.917187 14189 catalog_manager.cc:1347] Generated new cluster ID: bdf2398f06d84bdd885e37c591160ed3
I20260927 22:50:13.917291 14189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:50:13.934417 14189 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:50:13.935777 14189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:50:13.941679 14189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fdf094dd4f040c49630dd5ca87217f8: Generated new TSK 0
I20260927 22:50:13.942526 14189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:50:13.952277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14110
I20260927 22:50:13.953337 14172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:14.043892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14110
2026-09-27T22:50:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1811 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:50:14 dist-test-slave-kr25 krb5kdc[14205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:50:14 dist-test-slave-kr25 krb5kdc[14205](info): set up 2 sockets
Sep 27 22:50:14 dist-test-slave-kr25 krb5kdc[14205](info): commencing operation
krb5kdc: starting...
W20260927 22:50:15.047665 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:50:15 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:50:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:50:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:50:15.259255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:15.568234 14218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:15.568518 14218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:15.568636 14218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:15.575618 14218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:15.575722 14218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:15.575796 14218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:15.575882 14218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:15.575928 14218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:15.585624 14218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:15.586861 14218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:15.588081 14218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:15.598922 14223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:15.598903 14224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:15.598963 14226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:15.599684 14218 server_base.cc:1034] running on GCE node
I20260927 22:50:15.600436 14218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:15.601761 14218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:15.602962 14218 hybrid_clock.cc:648] HybridClock initialized: now 1790549415602891 us; error 46 us; skew 500 ppm
I20260927 22:50:15.603269 14218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:15 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:15.611027 14218 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:50:15.612402 14218 webserver.cc:460] Webserver started at http://127.0.97.62:44449/ using document root <none> and password file <none>
I20260927 22:50:15.613046 14218 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:15.613160 14218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:15.613432 14218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:15.616338 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "5d12937fe9554537b99eec1167aaa3f3"
format_stamp: "Formatted at 2026-09-27 22:50:15 on dist-test-slave-kr25"
server_key: "597fc14b1da3b55888fee6ab165588f1"
server_key_iv: "57ca5cc0066cdf921f7d104d8fec184d"
server_key_version: "encryptionkey@0"
I20260927 22:50:15.617054 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d12937fe9554537b99eec1167aaa3f3"
format_stamp: "Formatted at 2026-09-27 22:50:15 on dist-test-slave-kr25"
server_key: "597fc14b1da3b55888fee6ab165588f1"
server_key_iv: "57ca5cc0066cdf921f7d104d8fec184d"
server_key_version: "encryptionkey@0"
I20260927 22:50:15.622073 14218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:50:15.625416 14233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:15.626654 14218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:50:15.626809 14218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "5d12937fe9554537b99eec1167aaa3f3"
format_stamp: "Formatted at 2026-09-27 22:50:15 on dist-test-slave-kr25"
server_key: "597fc14b1da3b55888fee6ab165588f1"
server_key_iv: "57ca5cc0066cdf921f7d104d8fec184d"
server_key_version: "encryptionkey@0"
I20260927 22:50:15.626986 14218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:15.646039 14218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:15.649822 14218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:15.650054 14218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:15.666044 14218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45583
I20260927 22:50:15.666071 14284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45583 every 8 connection(s)
I20260927 22:50:15.667364 14218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:50:15.668812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14218
I20260927 22:50:15.669008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:50:15.669652 388 external_mini_cluster.cc:1428] Setting key 7355eb6137899f72a2d4cc813c7fa2db
I20260927 22:50:15.671350 14285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:50:15 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:15.684262 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Bootstrap starting.
I20260927 22:50:15.687420 14285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:15.688509 14285 log.cc:824] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:15.691069 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: No bootstrap required, opened a new log
I20260927 22:50:15.691848 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:15.671601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.0.97.62:45583)
0927 22:50:15.672314 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:15.672329 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:15.672387 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:50:15.674282 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:15.674308 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:15.674805 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:15.675258 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:15.676600 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:15.678574 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:15.679926 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:15.680894 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:15.681070 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:15.685088 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:15.685095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:15.685109 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:15.685141 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:15.688023 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:15.688773 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:15.688780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:15.688785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:15.688936 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:15.689292 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:15.689295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:15.689297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:15.689773 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:15.690037 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:15.690481 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 22:50:15.690876 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":286,"threads_started":1}
I20260927 22:50:15.698539 14285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:15.698938 14285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:15.699039 14285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d12937fe9554537b99eec1167aaa3f3, State: Initialized, Role: FOLLOWER
I20260927 22:50:15.699608 14285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:15.699828 14285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:50:15.699954 14285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:50:15.700099 14285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:15.702191 14285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:15.702646 14285 leader_election.cc:302] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d12937fe9554537b99eec1167aaa3f3; no voters:
I20260927 22:50:15.703248 14285 leader_election.cc:288] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:50:15.703361 14290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:15.704151 14290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 LEADER]: Becoming Leader. State: Replica: 5d12937fe9554537b99eec1167aaa3f3, State: Running, Role: LEADER
I20260927 22:50:15.704671 14290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:15.705449 14285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:15.707387 14291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d12937fe9554537b99eec1167aaa3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260927 22:50:15.707479 14292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d12937fe9554537b99eec1167aaa3f3. Latest consensus state: current_term: 1 leader_uuid: "5d12937fe9554537b99eec1167aaa3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260927 22:50:15.707823 14292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:15.707818 14291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:15.712471 14296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:50:15.717221 14296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:50:15.724699 14296 catalog_manager.cc:1347] Generated new cluster ID: 7c61a41934574a48a367ca60d68b273a
I20260927 22:50:15.724798 14296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:50:15.746620 14296 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:50:15.748515 14296 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:50:15.761752 14296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Generated new TSK 0
I20260927 22:50:15.762538 14296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:50:15.824945 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45583
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:16.170823 14311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:16.171080 14311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:16.171200 14311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:16.178436 14311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:16.178555 14311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:16.178699 14311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:50:16.188597 14311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:16.189870 14311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:16.191241 14311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:16.203646 14317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:16.203678 14319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:16.203699 14316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:16.204990 14311 server_base.cc:1034] running on GCE node
I20260927 22:50:16.205533 14311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:16.206429 14311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:16.207619 14311 hybrid_clock.cc:648] HybridClock initialized: now 1790549416207553 us; error 54 us; skew 500 ppm
I20260927 22:50:16.207904 14311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:16 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:16.210765 14311 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:50:16.212136 14311 webserver.cc:460] Webserver started at http://127.0.97.1:38399/ using document root <none> and password file <none>
I20260927 22:50:16.212764 14311 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:16.212872 14311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:16.213164 14311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:16.215992 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "108029489b8f4b2f865eef7d2bb07966"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "598fb0845a1c2d69fb0972ddb6668573"
server_key_iv: "af6f6c99bf02bd427c7d1b06fab361e8"
server_key_version: "encryptionkey@0"
I20260927 22:50:16.216709 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "108029489b8f4b2f865eef7d2bb07966"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "598fb0845a1c2d69fb0972ddb6668573"
server_key_iv: "af6f6c99bf02bd427c7d1b06fab361e8"
server_key_version: "encryptionkey@0"
I20260927 22:50:16.221647 14311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:50:16.225062 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:16.226323 14311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:50:16.226471 14311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "108029489b8f4b2f865eef7d2bb07966"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "598fb0845a1c2d69fb0972ddb6668573"
server_key_iv: "af6f6c99bf02bd427c7d1b06fab361e8"
server_key_version: "encryptionkey@0"
I20260927 22:50:16.226649 14311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:16.251173 14311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:16.255270 14311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:16.255563 14311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:16.256532 14311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:16.257833 14311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:16.257911 14311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:16.258023 14311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:16.258077 14311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:16.287117 14311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36681
I20260927 22:50:16.287155 14438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36681 every 8 connection(s)
I20260927 22:50:16.288359 14311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:50:16.289036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14311
I20260927 22:50:16.289235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:50:16.289943 388 external_mini_cluster.cc:1428] Setting key 73a59aae70360743d12358f79c4caf59
Sep 27 22:50:16 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:16.308790 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:16.290876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56041 (local address 127.0.97.62:45583)
0927 22:50:16.291303 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:16.291307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:16.292369 (+ 1062us) server_negotiation.cc:407] Connection header received
0927 22:50:16.294112 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:16.294117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:16.294195 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:16.294296 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:16.295889 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.296599 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.298341 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.298735 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.298838 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:16.302669 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:16.302675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:16.302680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:16.302721 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:16.305725 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:16.306396 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:16.306401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:16.306405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:16.306549 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:16.306949 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:16.306953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:16.306956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:16.307319 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:16.307472 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:16.308175 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 22:50:16.308420 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260927 22:50:16.309508 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:16.291500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:56041)
0927 22:50:16.292233 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:16.292269 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:16.293857 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:16.294556 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:16.294598 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:16.295133 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:16.295711 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:16.295730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.296743 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.296747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:16.298167 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:16.298182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.298953 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.298963 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:16.299222 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:16.299928 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:16.299957 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:16.302432 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:16.305868 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:16.305874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:16.305887 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:16.306278 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:16.306648 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:16.306651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:16.306654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:16.306825 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:16.307466 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:16.307475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:16.307786 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 22:50:16.308016 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 22:50:16.308399 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":286,"threads_started":1}
I20260927 22:50:16.310979 14439 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45583
I20260927 22:50:16.311252 14439 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:50:16.311971 14439 heartbeater.cc:511] Master 127.0.97.62:45583 requested a full tablet report, sending...
I20260927 22:50:16.313954 14250 ts_manager.cc:194] Registered new tserver with Master: 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:50:16.315848 14250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:50:16.340758 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45583
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:16.654034 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:16.654358 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:16.654456 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:16.661563 14445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:16.661688 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:16.661813 14445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:50:16.675319 14445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:16.676893 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:16.678411 14445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:16.692423 14453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:16.692447 14451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:16.692478 14450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:16.693243 14445 server_base.cc:1034] running on GCE node
I20260927 22:50:16.693758 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:16.694669 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:16.695856 14445 hybrid_clock.cc:648] HybridClock initialized: now 1790549416695794 us; error 55 us; skew 500 ppm
I20260927 22:50:16.696128 14445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:16 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:16.699090 14445 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:50:16.700584 14445 webserver.cc:460] Webserver started at http://127.0.97.2:46337/ using document root <none> and password file <none>
I20260927 22:50:16.701243 14445 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:16.701347 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:16.701627 14445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:16.704459 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "45136f324a8247d2bccaa028faac1d36"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "6b2dd35ce6d707e3f3ebbf9f9b1ea743"
server_key_iv: "d28d4631bf847063351f60cf367389f7"
server_key_version: "encryptionkey@0"
I20260927 22:50:16.705188 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "45136f324a8247d2bccaa028faac1d36"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "6b2dd35ce6d707e3f3ebbf9f9b1ea743"
server_key_iv: "d28d4631bf847063351f60cf367389f7"
server_key_version: "encryptionkey@0"
I20260927 22:50:16.710104 14445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:50:16.713382 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:16.714627 14445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:50:16.714748 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "45136f324a8247d2bccaa028faac1d36"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "6b2dd35ce6d707e3f3ebbf9f9b1ea743"
server_key_iv: "d28d4631bf847063351f60cf367389f7"
server_key_version: "encryptionkey@0"
I20260927 22:50:16.714905 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:16.727108 14445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:16.730949 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:16.731187 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:16.732201 14445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:16.733417 14445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:16.733491 14445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:16.733582 14445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:16.733625 14445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:16.762717 14445 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40347
I20260927 22:50:16.762784 14572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40347 every 8 connection(s)
I20260927 22:50:16.763945 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:50:16.770833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14445
I20260927 22:50:16.771032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:50:16.771528 388 external_mini_cluster.cc:1428] Setting key 4107f976ccfd2dc9d9c195b5b1348d69
Sep 27 22:50:16 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:16.783708 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:16.766309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54105 (local address 127.0.97.62:45583)
0927 22:50:16.766485 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:16.766490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:16.767592 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 22:50:16.769377 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:16.769381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:16.769453 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:16.769541 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:16.771215 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.772074 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.773746 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.774199 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.774332 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:16.777888 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:16.777893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:16.777896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:16.777931 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:16.780408 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:16.781191 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:16.781195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:16.781199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:16.781375 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:16.781947 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:16.781952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:16.781954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:16.782240 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:16.782392 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:16.783213 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 22:50:16.783411 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:50:16.785059 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:16.766731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:54105)
0927 22:50:16.767406 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:16.767481 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:16.769109 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:16.769735 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:16.769743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:16.770260 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:16.771013 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:16.771043 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.772255 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.772259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:16.773590 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:16.773604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:16.774446 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:16.774454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:16.774677 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:16.775406 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:16.775428 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:16.777651 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:16.780617 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:16.780625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:16.780640 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:16.781077 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:16.781487 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:16.781492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:16.781495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:16.781747 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:16.782394 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:16.782402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:16.782817 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 22:50:16.783097 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 22:50:16.783737 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":268,"threads_started":1}
I20260927 22:50:16.786566 14573 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45583
I20260927 22:50:16.786901 14573 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:16.787741 14573 heartbeater.cc:511] Master 127.0.97.62:45583 requested a full tablet report, sending...
I20260927 22:50:16.789268 14250 ts_manager.cc:194] Registered new tserver with Master: 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347)
I20260927 22:50:16.790103 14250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54105
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:50:16.822106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45583
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:17.134840 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:17.135146 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:17.135300 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:17.142520 14578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:17.142650 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:17.142809 14578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:50:17.152712 14578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:17.154028 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:17.155370 14578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:17.168097 14584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:17.168071 14583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:17.168208 14586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:17.168640 14578 server_base.cc:1034] running on GCE node
I20260927 22:50:17.169262 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:17.170167 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:17.171367 14578 hybrid_clock.cc:648] HybridClock initialized: now 1790549417171307 us; error 44 us; skew 500 ppm
I20260927 22:50:17.171692 14578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:17 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:17.174784 14578 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:50:17.176307 14578 webserver.cc:460] Webserver started at http://127.0.97.3:39305/ using document root <none> and password file <none>
I20260927 22:50:17.177029 14578 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:17.177146 14578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:17.177439 14578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:17.180457 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "92735147983d4ece810814e55aadc504"
format_stamp: "Formatted at 2026-09-27 22:50:17 on dist-test-slave-kr25"
server_key: "9f89fd527e8ff006a2b8d6902a3ff90c"
server_key_iv: "9ccff55aabaa07d74971abee81e37757"
server_key_version: "encryptionkey@0"
I20260927 22:50:17.181229 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "92735147983d4ece810814e55aadc504"
format_stamp: "Formatted at 2026-09-27 22:50:17 on dist-test-slave-kr25"
server_key: "9f89fd527e8ff006a2b8d6902a3ff90c"
server_key_iv: "9ccff55aabaa07d74971abee81e37757"
server_key_version: "encryptionkey@0"
I20260927 22:50:17.186342 14578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:50:17.189800 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:17.191062 14578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:50:17.191208 14578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "92735147983d4ece810814e55aadc504"
format_stamp: "Formatted at 2026-09-27 22:50:17 on dist-test-slave-kr25"
server_key: "9f89fd527e8ff006a2b8d6902a3ff90c"
server_key_iv: "9ccff55aabaa07d74971abee81e37757"
server_key_version: "encryptionkey@0"
I20260927 22:50:17.191387 14578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:17.211645 14578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:17.215873 14578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:17.216122 14578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:17.216995 14578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:17.218267 14578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:17.218340 14578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:17.218430 14578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:17.218470 14578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:17.245667 14578 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41759
I20260927 22:50:17.245724 14705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41759 every 8 connection(s)
I20260927 22:50:17.246892 14578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:50:17.252776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14578
I20260927 22:50:17.252943 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:50:17.253402 388 external_mini_cluster.cc:1428] Setting key b5a3d77854a5da2c8892fcba0015d326
Sep 27 22:50:17 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:17.266217 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.249601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33655 (local address 127.0.97.62:45583)
0927 22:50:17.249780 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.249787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.250845 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 22:50:17.252570 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.252575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.252655 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.252742 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:17.254348 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.255209 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.256972 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.257374 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.257480 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.260765 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:17.260770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:17.260774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:17.260811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:17.263314 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.264012 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.264015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.264018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.264147 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.264498 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.264501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.264503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.264751 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:17.264854 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.265650 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.265889 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:50:17.267251 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.249962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:33655)
0927 22:50:17.250666 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:17.250703 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:17.252299 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:17.252998 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:17.253010 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:17.253550 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:17.254185 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.254204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.255465 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.255471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.256825 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.256839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.257574 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.257579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.257778 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.258360 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:17.258387 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:17.260552 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:17.263467 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:17.263477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:17.263494 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:17.263915 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:17.264225 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:17.264229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:17.264231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:17.264418 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:17.264860 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:17.264872 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:17.265201 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 22:50:17.265487 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:50:17.266016 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
I20260927 22:50:17.268738 14706 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45583
I20260927 22:50:17.269037 14706 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:17.269762 14706 heartbeater.cc:511] Master 127.0.97.62:45583 requested a full tablet report, sending...
I20260927 22:50:17.271036 14250 ts_manager.cc:194] Registered new tserver with Master: 92735147983d4ece810814e55aadc504 (127.0.97.3:41759)
I20260927 22:50:17.271872 14250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33655
I20260927 22:50:17.281051 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:50:17.297506 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.285219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.0.97.62:45583)
0927 22:50:17.285369 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.285376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.285807 (+ 431us) server_negotiation.cc:407] Connection header received
0927 22:50:17.285980 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.285985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.286046 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.286126 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:17.287601 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.288309 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.289589 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.289990 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.290098 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.292425 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:17.292430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:17.292434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:17.292465 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:17.294951 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.295659 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.295664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.295666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.295787 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.296139 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.296143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.296145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.296467 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:17.296702 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.297002 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.297214 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:50:17.301487 14250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:50:17.308166 14250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:50:17.319229 14439 heartbeater.cc:503] Master 127.0.97.62:45583 was elected leader, sending a full tablet report...
I20260927 22:50:17.332044 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.321068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41759 (local address 127.0.0.1:38370)
0927 22:50:17.321875 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:17.321919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:17.322107 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:17.322769 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:17.322786 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:17.322828 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:17.323286 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.325346 (+ 2060us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.327173 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.327177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.329185 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.329197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.330898 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.330906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.331102 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.331159 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:50:17.331321 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:50:17.331535 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":63,"thread_start_us":1498,"threads_started":1}
I20260927 22:50:17.332715 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.321624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36681 (local address 127.0.0.1:58404)
0927 22:50:17.323399 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:17.323419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:17.323662 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:17.324678 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:17.324684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:17.324714 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:17.325153 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.325166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.326977 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.326982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.329635 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.329651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.331833 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.331839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.332021 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.332058 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:50:17.332142 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:50:17.332259 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"mutex_wait_us":246,"thread_start_us":1828,"threads_started":1}
I20260927 22:50:17.332918 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.321431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.0.97.3:41759)
0927 22:50:17.322188 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.322199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.322236 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:50:17.322357 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.322366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.322569 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.322744 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:17.325547 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.327038 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.329329 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.330663 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.330828 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.331066 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.331362 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.331556 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":398,"threads_started":1}
I20260927 22:50:17.333424 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.323308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.0.1:54258)
0927 22:50:17.323828 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:17.323844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:17.323950 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:17.326304 (+ 2354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:17.326308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:17.326334 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:17.326678 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.326687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.328570 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.328576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.330481 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.330494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.332642 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.332650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.332841 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.332875 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:50:17.332961 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:50:17.333084 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":278,"threads_started":1}
I20260927 22:50:17.333591 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.322145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.0.97.1:36681)
0927 22:50:17.322705 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.322712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.323540 (+ 828us) server_negotiation.cc:407] Connection header received
0927 22:50:17.323908 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.323918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.324191 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.324376 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:17.325316 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.326852 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.329811 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.331343 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.331480 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.331695 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.333064 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.333234 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":262,"threads_started":1}
I20260927 22:50:17.334167 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.323557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.0.97.2:40347)
0927 22:50:17.325856 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.325865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.325891 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:50:17.325965 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.325977 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.326131 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.326303 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:17.326844 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.328378 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.330632 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.332164 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.332292 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.332469 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.333647 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.333832 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2162,"thread_start_us":2045,"threads_started":1}
I20260927 22:50:17.336820 14641 tablet_service.cc:1467] Processing CreateTablet for tablet a241c6a7754b4396bb06fcdd0c10c7c0 (DEFAULT_TABLE table=test-table [id=82ee660be0914af28f1ad7cc0d8f1b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:17.337121 14374 tablet_service.cc:1467] Processing CreateTablet for tablet a241c6a7754b4396bb06fcdd0c10c7c0 (DEFAULT_TABLE table=test-table [id=82ee660be0914af28f1ad7cc0d8f1b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:17.337512 14508 tablet_service.cc:1467] Processing CreateTablet for tablet a241c6a7754b4396bb06fcdd0c10c7c0 (DEFAULT_TABLE table=test-table [id=82ee660be0914af28f1ad7cc0d8f1b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:17.338392 14641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a241c6a7754b4396bb06fcdd0c10c7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:17.338394 14374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a241c6a7754b4396bb06fcdd0c10c7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:17.338775 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a241c6a7754b4396bb06fcdd0c10c7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:17.351351 14724 tablet_bootstrap.cc:492] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Bootstrap starting.
I20260927 22:50:17.352645 14725 tablet_bootstrap.cc:492] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Bootstrap starting.
I20260927 22:50:17.354108 14726 tablet_bootstrap.cc:492] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Bootstrap starting.
I20260927 22:50:17.354807 14724 tablet_bootstrap.cc:654] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:17.356017 14725 tablet_bootstrap.cc:654] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:17.356101 14724 log.cc:824] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:17.357224 14726 tablet_bootstrap.cc:654] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:17.357239 14725 log.cc:824] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:17.358301 14726 log.cc:824] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:17.359313 14724 tablet_bootstrap.cc:492] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: No bootstrap required, opened a new log
I20260927 22:50:17.359642 14724 ts_tablet_manager.cc:1397] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:50:17.360103 14725 tablet_bootstrap.cc:492] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: No bootstrap required, opened a new log
I20260927 22:50:17.360363 14725 ts_tablet_manager.cc:1397] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:50:17.361174 14726 tablet_bootstrap.cc:492] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: No bootstrap required, opened a new log
I20260927 22:50:17.361392 14726 ts_tablet_manager.cc:1397] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 22:50:17.368194 14726 raft_consensus.cc:348] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.368541 14726 raft_consensus.cc:374] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:17.368661 14726 raft_consensus.cc:729] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92735147983d4ece810814e55aadc504, State: Initialized, Role: FOLLOWER
I20260927 22:50:17.369231 14726 consensus_queue.cc:260] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.369665 14724 raft_consensus.cc:348] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.370093 14724 raft_consensus.cc:374] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:17.370177 14724 raft_consensus.cc:729] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45136f324a8247d2bccaa028faac1d36, State: Initialized, Role: FOLLOWER
I20260927 22:50:17.370229 14725 raft_consensus.cc:348] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.370601 14725 raft_consensus.cc:374] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:17.370682 14725 raft_consensus.cc:729] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108029489b8f4b2f865eef7d2bb07966, State: Initialized, Role: FOLLOWER
I20260927 22:50:17.370810 14724 consensus_queue.cc:260] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.371279 14725 consensus_queue.cc:260] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.371913 14706 heartbeater.cc:503] Master 127.0.97.62:45583 was elected leader, sending a full tablet report...
I20260927 22:50:17.372485 14726 ts_tablet_manager.cc:1428] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260927 22:50:17.373780 14573 heartbeater.cc:503] Master 127.0.97.62:45583 was elected leader, sending a full tablet report...
I20260927 22:50:17.374457 14724 ts_tablet_manager.cc:1428] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260927 22:50:17.376837 14725 ts_tablet_manager.cc:1428] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260927 22:50:17.498540 14707 tablet.cc:2369] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:50:17.515825 14574 tablet.cc:2369] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:50:17.540751 14440 tablet.cc:2369] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:17.601307 14730 raft_consensus.cc:482] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:50:17.601634 14730 raft_consensus.cc:504] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.603312 14730 leader_election.cc:288] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681), 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347)
I20260927 22:50:17.608726 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.603787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36681 (local address 127.0.97.3:41133)
0927 22:50:17.604083 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:17.604098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:17.604228 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:17.604547 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:17.604551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:17.604576 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:17.604737 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.604745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.605730 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.605736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.607066 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.607077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.608194 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.608204 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.608344 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.608369 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:50:17.608440 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:50:17.608526 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":3}
I20260927 22:50:17.608839 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.603773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41133 (local address 127.0.97.1:36681)
0927 22:50:17.603992 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.603998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.604184 (+ 186us) server_negotiation.cc:407] Connection header received
0927 22:50:17.604360 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.604366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.604433 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.604508 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:17.604873 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.605599 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.607214 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.608059 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.608187 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.608287 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.608480 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.608604 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:50:17.609756 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.603780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44191 (local address 127.0.97.2:40347)
0927 22:50:17.603987 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.603993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.604633 (+ 640us) server_negotiation.cc:407] Connection header received
0927 22:50:17.604847 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.604851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.604909 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.604983 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:17.605491 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.606174 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.607987 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.609033 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.609159 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.609242 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.609501 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.609590 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:50:17.609946 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.603814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.97.3:44191)
0927 22:50:17.604525 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:17.604538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:17.604719 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:17.605008 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:17.605012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:17.605033 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:17.605320 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.605328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.606322 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.606327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.607816 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:17.607831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.609203 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.609208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:17.609357 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.609390 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:50:17.609482 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:50:17.609593 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"spinlock_wait_cycles":24320,"thread_start_us":377,"threads_started":1}
I20260927 22:50:17.609787 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a241c6a7754b4396bb06fcdd0c10c7c0" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108029489b8f4b2f865eef7d2bb07966" is_pre_election: true
I20260927 22:50:17.610291 14394 raft_consensus.cc:2454] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 0.
I20260927 22:50:17.610978 14597 leader_election.cc:302] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 108029489b8f4b2f865eef7d2bb07966, 92735147983d4ece810814e55aadc504; no voters:
I20260927 22:50:17.610895 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a241c6a7754b4396bb06fcdd0c10c7c0" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45136f324a8247d2bccaa028faac1d36" is_pre_election: true
I20260927 22:50:17.611374 14730 raft_consensus.cc:2781] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:50:17.611467 14528 raft_consensus.cc:2454] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 0.
I20260927 22:50:17.611562 14730 raft_consensus.cc:482] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:50:17.611662 14730 raft_consensus.cc:3037] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:17.614507 14730 raft_consensus.cc:504] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.615111 14730 leader_election.cc:288] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 election: Requested vote from peers 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681), 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347)
I20260927 22:50:17.615624 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a241c6a7754b4396bb06fcdd0c10c7c0" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108029489b8f4b2f865eef7d2bb07966"
I20260927 22:50:17.615612 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a241c6a7754b4396bb06fcdd0c10c7c0" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45136f324a8247d2bccaa028faac1d36"
I20260927 22:50:17.615816 14394 raft_consensus.cc:3037] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:17.615816 14528 raft_consensus.cc:3037] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:17.618897 14528 raft_consensus.cc:2454] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 1.
I20260927 22:50:17.618897 14394 raft_consensus.cc:2454] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 1.
I20260927 22:50:17.619346 14597 leader_election.cc:302] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 108029489b8f4b2f865eef7d2bb07966, 92735147983d4ece810814e55aadc504; no voters:
I20260927 22:50:17.619640 14730 raft_consensus.cc:2781] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:17.620023 14730 raft_consensus.cc:686] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 1 LEADER]: Becoming Leader. State: Replica: 92735147983d4ece810814e55aadc504, State: Running, Role: LEADER
I20260927 22:50:17.620445 14730 consensus_queue.cc:237] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260927 22:50:17.624975 14248 catalog_manager.cc:5495] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92735147983d4ece810814e55aadc504 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "92735147983d4ece810814e55aadc504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } health_report { overall_health: UNKNOWN } } }
I20260927 22:50:17.696827 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.691463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.97.3:41759)
0927 22:50:17.691646 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.691652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.691754 (+ 102us) server_negotiation.cc:407] Connection header received
0927 22:50:17.691960 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.691964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.692040 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.692168 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:50:17.692545 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.693260 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.694627 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.695085 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.695191 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.695745 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:50:17.696275 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:50:17.696415 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.696501 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.696581 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:50:17.704764 14394 raft_consensus.cc:1261] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Refusing update from remote peer 92735147983d4ece810814e55aadc504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:17.704764 14528 raft_consensus.cc:1261] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Refusing update from remote peer 92735147983d4ece810814e55aadc504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:17.705878 14734 consensus_queue.cc:1035] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:17.706207 14730 consensus_queue.cc:1035] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:17.718078 14739 mvcc.cc:205] Tried to move back new op lower bound from 7334090414908661760 to 7334090414576771072. Current Snapshot: MvccSnapshot[applied={T|T < 7334090414908661760}]
I20260927 22:50:17.718380 14738 mvcc.cc:205] Tried to move back new op lower bound from 7334090414908661760 to 7334090414576771072. Current Snapshot: MvccSnapshot[applied={T|T < 7334090414908661760}]
I20260927 22:50:17.732016 14248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:17.732251 14248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:17.735608 14248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Sending DeleteTablet for 3 replicas of tablet a241c6a7754b4396bb06fcdd0c10c7c0
I20260927 22:50:17.736598 14374 tablet_service.cc:1514] Processing DeleteTablet for tablet a241c6a7754b4396bb06fcdd0c10c7c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58404
I20260927 22:50:17.736963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14311
I20260927 22:50:17.736927 14641 tablet_service.cc:1514] Processing DeleteTablet for tablet a241c6a7754b4396bb06fcdd0c10c7c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38370
I20260927 22:50:17.737567 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet a241c6a7754b4396bb06fcdd0c10c7c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54258
I20260927 22:50:17.754207 14747 tablet_replica.cc:330] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: stopping tablet replica
I20260927 22:50:17.754599 14747 raft_consensus.cc:2229] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:17.755070 14747 raft_consensus.cc:2258] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:17.758342 14747 ts_tablet_manager.cc:1905] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:17.766013 14747 ts_tablet_manager.cc:1918] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:17.766163 14747 log.cc:1196] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/a241c6a7754b4396bb06fcdd0c10c7c0
I20260927 22:50:17.767201 14747 ts_tablet_manager.cc:1939] T a241c6a7754b4396bb06fcdd0c10c7c0 P 108029489b8f4b2f865eef7d2bb07966: Deleting consensus metadata
I20260927 22:50:17.769160 14237 catalog_manager.cc:4841] TS 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681): tablet a241c6a7754b4396bb06fcdd0c10c7c0 (table test-table [id=82ee660be0914af28f1ad7cc0d8f1b4a]) successfully deleted
I20260927 22:50:17.770097 14750 tablet_replica.cc:330] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: stopping tablet replica
I20260927 22:50:17.770503 14750 raft_consensus.cc:2229] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:17.771034 14750 raft_consensus.cc:2258] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:17.772221 14748 tablet_replica.cc:330] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: stopping tablet replica
I20260927 22:50:17.772755 14748 raft_consensus.cc:2229] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:50:17.773456 14748 raft_consensus.cc:2258] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:17.775391 14750 ts_tablet_manager.cc:1905] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:17.775465 14748 ts_tablet_manager.cc:1905] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:50:17 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:50:17.782891 14748 ts_tablet_manager.cc:1918] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:17.783154 14748 log.cc:1196] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/a241c6a7754b4396bb06fcdd0c10c7c0
I20260927 22:50:17.783798 14748 ts_tablet_manager.cc:1939] T a241c6a7754b4396bb06fcdd0c10c7c0 P 92735147983d4ece810814e55aadc504: Deleting consensus metadata
I20260927 22:50:17.785687 14234 catalog_manager.cc:4841] TS 92735147983d4ece810814e55aadc504 (127.0.97.3:41759): tablet a241c6a7754b4396bb06fcdd0c10c7c0 (table test-table [id=82ee660be0914af28f1ad7cc0d8f1b4a]) successfully deleted
I20260927 22:50:17.790324 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.738211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.0.97.1:36681)
0927 22:50:17.738395 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.738401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.771898 (+ 33497us) server_negotiation.cc:407] Connection header received
0927 22:50:17.772183 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.772190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.772272 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.772346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:17.774247 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.775314 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.777277 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.777858 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.777953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.783329 (+ 5376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:17.783364 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:17.783387 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:17.783449 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:17.787119 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.788006 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.788016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.788020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.788176 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.788646 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.788652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.788655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.789111 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:17.789436 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.789751 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.789968 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:50:17.790634 14750 ts_tablet_manager.cc:1918] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:17.790788 14750 log.cc:1196] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/a241c6a7754b4396bb06fcdd0c10c7c0
I20260927 22:50:17.791000 14434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:17.791242 14750 ts_tablet_manager.cc:1939] T a241c6a7754b4396bb06fcdd0c10c7c0 P 45136f324a8247d2bccaa028faac1d36: Deleting consensus metadata
I20260927 22:50:17.793010 14234 catalog_manager.cc:4841] TS 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347): tablet a241c6a7754b4396bb06fcdd0c10c7c0 (table test-table [id=82ee660be0914af28f1ad7cc0d8f1b4a]) successfully deleted
I20260927 22:50:17.878568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14311
I20260927 22:50:17.899591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14445
Sep 27 22:50:17 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:50:17.916007 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:17.900357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.0.97.2:40347)
0927 22:50:17.900550 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:17.900556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:17.900578 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:50:17.900718 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:17.900725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:17.900810 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:17.900922 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:17.902189 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.903267 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.904693 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:17.905350 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:17.905471 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:17.909674 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:17.909698 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:17.909718 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:17.909759 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:17.913296 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.913944 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.913954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.913958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.914107 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:17.914452 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:17.914457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:17.914460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:17.914908 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:17.915213 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:17.915382 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 22:50:17.915629 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:50:17.916708 14568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:18.003568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14445
I20260927 22:50:18.024082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14578
Sep 27 22:50:18 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:50:18.042367 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:18.024822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.0.97.3:41759)
0927 22:50:18.024990 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:18.024994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:18.025011 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:50:18.025262 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:18.025267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:18.025383 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:18.025482 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:18.027182 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:18.028208 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:18.030096 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:18.030601 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:18.030716 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:18.036466 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:18.036482 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:18.036499 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:18.036538 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:18.039573 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:18.040397 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:18.040405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:18.040407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:18.040567 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:18.040938 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:18.040941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:18.040943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:18.041270 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:18.041525 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:18.041848 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 22:50:18.042002 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:50:18.043004 14701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:18.165881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14578
I20260927 22:50:18.181348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14218
I20260927 22:50:18.182606 14280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:18.290130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14218
I20260927 22:50:18.307518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--webserver_interface=127.0.97.62
--webserver_port=44449
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:18.620805 14755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:18.621115 14755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:18.621245 14755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:18.628328 14755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:18.628441 14755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:18.628515 14755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:18.628588 14755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:18.628648 14755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:18.638342 14755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:18.639596 14755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:18.640875 14755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:18.661386 14761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:18.661540 14764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:18.661521 14762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:18.662205 14755 server_base.cc:1034] running on GCE node
I20260927 22:50:18.663102 14755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:18.665748 14755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:18.667011 14755 hybrid_clock.cc:648] HybridClock initialized: now 1790549418666874 us; error 95 us; skew 500 ppm
I20260927 22:50:18.667574 14755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:18 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:18.671211 14755 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:50:18.672933 14755 webserver.cc:460] Webserver started at http://127.0.97.62:44449/ using document root <none> and password file <none>
I20260927 22:50:18.673688 14755 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:18.673820 14755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:18.679771 14755 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 22:50:18.682807 14771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:18.684280 14755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:50:18.684430 14755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "5d12937fe9554537b99eec1167aaa3f3"
format_stamp: "Formatted at 2026-09-27 22:50:15 on dist-test-slave-kr25"
server_key: "597fc14b1da3b55888fee6ab165588f1"
server_key_iv: "57ca5cc0066cdf921f7d104d8fec184d"
server_key_version: "encryptionkey@0"
I20260927 22:50:18.685453 14755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:18.702525 14755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:18.706889 14755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:18.707165 14755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:18.725004 14755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45583
I20260927 22:50:18.725023 14822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45583 every 8 connection(s)
I20260927 22:50:18.726468 14755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:50:18.728547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14755
I20260927 22:50:18.729485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36681
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45583
--webserver_port=38399
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:50:18.735618 14823 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:50:18.739598 14823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Bootstrap starting.
I20260927 22:50:18.759272 14823 log.cc:824] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:18.770816 14823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:50:18.771320 14823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Bootstrap complete.
I20260927 22:50:18.780241 14823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:18.781090 14823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d12937fe9554537b99eec1167aaa3f3, State: Initialized, Role: FOLLOWER
I20260927 22:50:18.781721 14823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:18.781960 14823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:50:18.782079 14823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:50:18.782227 14823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:50:18.786142 14823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:18.786659 14823 leader_election.cc:302] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d12937fe9554537b99eec1167aaa3f3; no voters:
I20260927 22:50:18.787230 14823 leader_election.cc:288] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:50:18.787420 14827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:50:18.788390 14827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 LEADER]: Becoming Leader. State: Replica: 5d12937fe9554537b99eec1167aaa3f3, State: Running, Role: LEADER
I20260927 22:50:18.788955 14827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:18.789486 14823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:18.791409 14829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d12937fe9554537b99eec1167aaa3f3. Latest consensus state: current_term: 2 leader_uuid: "5d12937fe9554537b99eec1167aaa3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260927 22:50:18.791484 14828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5d12937fe9554537b99eec1167aaa3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260927 22:50:18.791792 14829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:18.791836 14828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:18.792603 14833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:50:18.810096 14833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:50:18.811595 14833 catalog_manager.cc:1259] Loaded cluster ID: 7c61a41934574a48a367ca60d68b273a
I20260927 22:50:18.811709 14833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:50:18.814900 14833 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 22:50:18.834153 14833 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 22:50:18.847821 14833 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:50:18.849120 14833 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Loaded TSK: 0
I20260927 22:50:18.850252 14833 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:50:19.084739 14825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:19.085021 14825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:19.085151 14825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:19.092746 14825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:19.092870 14825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:19.093019 14825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:50:19.102756 14825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38399
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:19.104143 14825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:19.105476 14825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:19.119371 14853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:19.119771 14851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:19.120141 14850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:19.120301 14825 server_base.cc:1034] running on GCE node
I20260927 22:50:19.120947 14825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:19.121842 14825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:19.123071 14825 hybrid_clock.cc:648] HybridClock initialized: now 1790549419122999 us; error 58 us; skew 500 ppm
I20260927 22:50:19.123399 14825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:19 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:19.126780 14825 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:50:19.128306 14825 webserver.cc:460] Webserver started at http://127.0.97.1:38399/ using document root <none> and password file <none>
I20260927 22:50:19.128996 14825 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:19.129122 14825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:19.134334 14825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:50:19.136925 14860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:19.138226 14825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:50:19.138372 14825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "108029489b8f4b2f865eef7d2bb07966"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "598fb0845a1c2d69fb0972ddb6668573"
server_key_iv: "af6f6c99bf02bd427c7d1b06fab361e8"
server_key_version: "encryptionkey@0"
I20260927 22:50:19.139272 14825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:19.156641 14825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:19.160760 14825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:19.161008 14825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:19.162048 14825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:19.163471 14825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:19.163570 14825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:19.163671 14825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:19.163744 14825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:19.198830 14825 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36681
I20260927 22:50:19.198909 14972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36681 every 8 connection(s)
I20260927 22:50:19.200357 14825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:50:19.203877 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14825
I20260927 22:50:19.204651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40347
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45583
--webserver_port=46337
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:50:19 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:19.226465 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:19.203194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:48949)
0927 22:50:19.204214 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:19.204268 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:19.206660 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:19.208177 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:19.208188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:19.208928 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:19.209701 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:19.209725 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.211504 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.211509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:19.213201 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:19.213214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.214193 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.214200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:19.214455 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:19.215180 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:19.215202 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:19.217605 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:19.222106 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:19.222114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:19.222127 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:19.222621 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:19.223185 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:19.223188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:19.223191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:19.223424 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:19.224469 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:19.224476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:19.224803 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 22:50:19.225033 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 22:50:19.225454 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":510,"threads_started":1}
I20260927 22:50:19.227085 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:19.203081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48949 (local address 127.0.97.62:45583)
0927 22:50:19.203807 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:19.203822 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:19.204415 (+ 593us) server_negotiation.cc:407] Connection header received
0927 22:50:19.207238 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:19.207252 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:19.207659 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:19.207960 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:19.209942 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.211326 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.213342 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.213975 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.214152 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:19.217921 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:19.217933 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:19.217951 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:19.218000 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:19.221924 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:19.222787 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:19.222798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:19.222807 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:19.223052 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:19.223575 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:19.223581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:19.223584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:19.224304 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:19.224528 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:19.225323 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 22:50:19.225799 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":337,"threads_started":1}
I20260927 22:50:19.230151 14973 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45583
I20260927 22:50:19.230494 14973 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:19.231406 14973 heartbeater.cc:511] Master 127.0.97.62:45583 requested a full tablet report, sending...
I20260927 22:50:19.233942 14788 ts_manager.cc:194] Registered new tserver with Master: 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681)
I20260927 22:50:19.236124 14788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48949
W20260927 22:50:19.522536 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:19.522796 14977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:19.522890 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:19.529929 14977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:19.530031 14977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:19.530151 14977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:50:19.539795 14977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46337
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:19.541003 14977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:19.542263 14977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:19.554667 14983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:19.554688 14985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:19.554623 14982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:19.555409 14977 server_base.cc:1034] running on GCE node
I20260927 22:50:19.555979 14977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:19.556880 14977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:19.558082 14977 hybrid_clock.cc:648] HybridClock initialized: now 1790549419558037 us; error 35 us; skew 500 ppm
I20260927 22:50:19.558455 14977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:19 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:19.561529 14977 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:50:19.562827 14977 webserver.cc:460] Webserver started at http://127.0.97.2:46337/ using document root <none> and password file <none>
I20260927 22:50:19.563656 14977 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:19.563783 14977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:19.568812 14977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:50:19.571482 14992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:19.572753 14977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:50:19.572912 14977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "45136f324a8247d2bccaa028faac1d36"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "6b2dd35ce6d707e3f3ebbf9f9b1ea743"
server_key_iv: "d28d4631bf847063351f60cf367389f7"
server_key_version: "encryptionkey@0"
I20260927 22:50:19.573801 14977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:19.585950 14977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:19.589874 14977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:19.590160 14977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:19.591238 14977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:19.592558 14977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:19.592651 14977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:19.592747 14977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:19.592821 14977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:19.621435 14977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40347
I20260927 22:50:19.621505 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40347 every 8 connection(s)
I20260927 22:50:19.622864 14977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:50:19.627980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 14977
I20260927 22:50:19.628712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41759
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45583
--webserver_port=39305
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:50:19 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:19.644258 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:19.625327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42793 (local address 127.0.97.62:45583)
0927 22:50:19.625509 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:19.625513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:19.626607 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 22:50:19.628876 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:19.628884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:19.628992 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:19.629085 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:19.630960 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.631695 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.633795 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.634230 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.634320 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:19.638170 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:19.638176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:19.638181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:19.638220 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:19.641084 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:19.641760 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:19.641765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:19.641768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:19.641917 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:19.642272 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:19.642276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:19.642279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:19.642609 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:19.642758 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:19.643422 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 22:50:19.643942 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:50:19.644912 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:19.625776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:42793)
0927 22:50:19.626448 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:19.626489 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:19.628550 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:19.629325 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:19.629337 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:19.630047 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:19.630780 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:19.630814 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.631848 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.631853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:19.633653 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:19.633667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:19.634439 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:19.634445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:19.634703 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:19.635392 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:19.635418 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:19.637960 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:19.641218 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:19.641224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:19.641237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:19.641649 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:19.642006 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:19.642009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:19.642011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:19.642181 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:19.642773 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:19.642779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:19.643093 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 22:50:19.643325 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:50:19.643799 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":251,"threads_started":1}
I20260927 22:50:19.646446 15105 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45583
I20260927 22:50:19.646721 15105 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:19.647390 15105 heartbeater.cc:511] Master 127.0.97.62:45583 requested a full tablet report, sending...
I20260927 22:50:19.648787 14788 ts_manager.cc:194] Registered new tserver with Master: 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347)
I20260927 22:50:19.649600 14788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42793
W20260927 22:50:19.942591 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:19.942891 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:19.943038 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:19.950323 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:19.950446 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:19.950600 15108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:50:19.960240 15108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39305
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:19.961570 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:19.962808 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:19.974952 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:19.974998 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:19.975020 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:19.975723 15108 server_base.cc:1034] running on GCE node
I20260927 22:50:19.976284 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:19.977180 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:19.978368 15108 hybrid_clock.cc:648] HybridClock initialized: now 1790549419978314 us; error 47 us; skew 500 ppm
I20260927 22:50:19.978645 15108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:19 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:19.981505 15108 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:50:19.982949 15108 webserver.cc:460] Webserver started at http://127.0.97.3:39305/ using document root <none> and password file <none>
I20260927 22:50:19.983654 15108 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:19.983788 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:19.989015 15108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 22:50:19.991722 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:19.993029 15108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:50:19.993168 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "92735147983d4ece810814e55aadc504"
format_stamp: "Formatted at 2026-09-27 22:50:17 on dist-test-slave-kr25"
server_key: "9f89fd527e8ff006a2b8d6902a3ff90c"
server_key_iv: "9ccff55aabaa07d74971abee81e37757"
server_key_version: "encryptionkey@0"
I20260927 22:50:19.994077 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:20.012391 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:20.016409 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:20.016642 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:20.017584 15108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:20.018851 15108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:20.018939 15108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:20.019070 15108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:20.019133 15108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:20.049459 15108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41759
I20260927 22:50:20.049543 15235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41759 every 8 connection(s)
I20260927 22:50:20.050848 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:50:20.052874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 15108
Sep 27 22:50:20 dist-test-slave-kr25 krb5kdc[14205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:20.069155 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.054156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.0.97.62:45583)
0927 22:50:20.054630 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.054637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.054653 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:50:20.054710 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.054714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.054778 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.054886 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:20.056531 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.057323 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.058887 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.059372 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.059516 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.062977 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:20.062985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:20.062990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:20.063026 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:20.066330 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.067200 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.067204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.067207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.067332 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.067784 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.067789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.067796 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.068099 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:20.068301 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.068659 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.068847 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":352,"thread_start_us":251,"threads_started":1}
I20260927 22:50:20.072394 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.053522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34829 (local address 127.0.97.62:45583)
0927 22:50:20.053736 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.053742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.055158 (+ 1416us) server_negotiation.cc:407] Connection header received
0927 22:50:20.057016 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.057021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.057097 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.057182 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:20.058623 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.059343 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.060942 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.061528 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.061646 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.066025 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:20.066030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:20.066034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:20.066080 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:20.069232 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.070036 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.070040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.070042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.070178 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.070655 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.070660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.070662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.070953 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:20.071067 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.072002 (+ 935us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.072159 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:50:20.073380 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.054008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:34829)
0927 22:50:20.055013 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:20.055070 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:20.056757 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:20.057424 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:20.057433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:20.057934 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:20.058458 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.058485 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.059523 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.059528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.060805 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.060818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.061812 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.061823 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.062132 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.062880 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:20.062910 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:20.065762 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:20.069431 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:20.069442 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:20.069470 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:20.069914 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:20.070279 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:20.070336 (+ 57us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:20.070339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:20.070546 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:20.071076 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:20.071085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:20.071578 (+ 493us) client_negotiation.cc:769] Sending connection context
0927 22:50:20.071863 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 22:50:20.072265 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":524,"threads_started":1}
I20260927 22:50:20.074633 15236 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45583
I20260927 22:50:20.074906 15236 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:20.075672 15236 heartbeater.cc:511] Master 127.0.97.62:45583 requested a full tablet report, sending...
I20260927 22:50:20.076984 14788 ts_manager.cc:194] Registered new tserver with Master: 92735147983d4ece810814e55aadc504 (127.0.97.3:41759)
I20260927 22:50:20.077862 14788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34829
I20260927 22:50:20.083967 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:50:20.090005 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.084828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.0.97.62:45583)
0927 22:50:20.085018 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.085024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.085295 (+ 271us) server_negotiation.cc:407] Connection header received
0927 22:50:20.085478 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.085481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.085611 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.085720 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:50:20.086057 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.086741 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.088152 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.088574 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.088659 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.089070 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:50:20.089525 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:50:20.089618 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.089708 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.089764 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:50:20.090849 14788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:50:20.098553 14788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:50:20.121479 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.110134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36681 (local address 127.0.0.1:58416)
0927 22:50:20.111075 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:20.111120 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:20.111289 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:20.112497 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:20.112505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:20.112540 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:20.112975 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.112987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.115307 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.115314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.117837 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.117853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.120501 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.120507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.120719 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.120775 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:50:20.120899 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:50:20.121089 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":9744,"threads_started":1}
I20260927 22:50:20.122812 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.111011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.0.97.1:36681)
0927 22:50:20.111834 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.111844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.111884 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:50:20.111986 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.111995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.112256 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.113185 (+ 929us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:20.113457 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.115140 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.118072 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.119618 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.122013 (+ 2395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.122235 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.122303 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.122484 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":562,"threads_started":1}
I20260927 22:50:20.126734 14908 tablet_service.cc:1467] Processing CreateTablet for tablet 251d98ded52b4e68993fa68ee49785a9 (DEFAULT_TABLE table=test-table [id=ef971cb341e64987a64ed2221e7e2f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:20.128199 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251d98ded52b4e68993fa68ee49785a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:20.131698 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.111595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41759 (local address 127.0.0.1:38398)
0927 22:50:20.121731 (+ 10136us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:20.121749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:20.121847 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:20.124510 (+ 2663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:20.124516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:20.124544 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:20.124724 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.124734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.126960 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.126966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.129049 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.129065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.130926 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.130933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.131137 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.131189 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 22:50:20.131254 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 22:50:20.131399 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10045,"thread_start_us":19856,"threads_started":1}
I20260927 22:50:20.132679 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.111933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.0.97.3:41759)
0927 22:50:20.123974 (+ 12041us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.123983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.124013 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:50:20.124098 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.124105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.124305 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.124904 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:20.125169 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.126794 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.129264 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.130708 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.131928 (+ 1220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.132146 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.132213 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.132355 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11851,"thread_start_us":11710,"threads_started":1}
I20260927 22:50:20.133260 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.111391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.0.97.2:40347)
0927 22:50:20.124249 (+ 12858us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.124259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.124293 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:50:20.124384 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.124391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.124617 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.125409 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:20.125644 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.127056 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.129854 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.131198 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.132666 (+ 1468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.132877 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.132932 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.133037 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12699,"thread_start_us":12560,"threads_started":1}
I20260927 22:50:20.132705 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.110284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.0.1:54280)
0927 22:50:20.120152 (+ 9868us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:20.120172 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:20.120279 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:20.124837 (+ 4558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:20.124842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:20.124868 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:20.125212 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.125223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.127226 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.127234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.129658 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.129672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.131967 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.131976 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.132165 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.132196 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:50:20.132264 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 22:50:20.132386 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9737,"thread_start_us":369,"threads_started":1}
I20260927 22:50:20.136374 15171 tablet_service.cc:1467] Processing CreateTablet for tablet 251d98ded52b4e68993fa68ee49785a9 (DEFAULT_TABLE table=test-table [id=ef971cb341e64987a64ed2221e7e2f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:20.137518 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251d98ded52b4e68993fa68ee49785a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:20.137063 15040 tablet_service.cc:1467] Processing CreateTablet for tablet 251d98ded52b4e68993fa68ee49785a9 (DEFAULT_TABLE table=test-table [id=ef971cb341e64987a64ed2221e7e2f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:20.138414 15040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251d98ded52b4e68993fa68ee49785a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:20.144382 15252 tablet_bootstrap.cc:492] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Bootstrap starting.
I20260927 22:50:20.148099 15252 tablet_bootstrap.cc:654] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:20.149441 15252 log.cc:824] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:20.152294 15255 tablet_bootstrap.cc:492] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Bootstrap starting.
I20260927 22:50:20.152629 15252 tablet_bootstrap.cc:492] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: No bootstrap required, opened a new log
I20260927 22:50:20.152720 15254 tablet_bootstrap.cc:492] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Bootstrap starting.
I20260927 22:50:20.152889 15252 ts_tablet_manager.cc:1397] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 22:50:20.156070 15255 tablet_bootstrap.cc:654] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:20.156072 15254 tablet_bootstrap.cc:654] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:20.157261 15254 log.cc:824] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:20.157264 15255 log.cc:824] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:20.159960 15255 tablet_bootstrap.cc:492] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: No bootstrap required, opened a new log
I20260927 22:50:20.160202 15255 ts_tablet_manager.cc:1397] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:50:20.160254 15254 tablet_bootstrap.cc:492] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: No bootstrap required, opened a new log
I20260927 22:50:20.160508 15254 ts_tablet_manager.cc:1397] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260927 22:50:20.160951 15252 raft_consensus.cc:348] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.161396 15252 raft_consensus.cc:374] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:20.161485 15252 raft_consensus.cc:729] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108029489b8f4b2f865eef7d2bb07966, State: Initialized, Role: FOLLOWER
I20260927 22:50:20.162163 15252 consensus_queue.cc:260] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.163223 14973 heartbeater.cc:503] Master 127.0.97.62:45583 was elected leader, sending a full tablet report...
I20260927 22:50:20.165098 15252 ts_tablet_manager.cc:1428] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:50:20.168622 15255 raft_consensus.cc:348] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.169031 15255 raft_consensus.cc:374] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:20.168876 15254 raft_consensus.cc:348] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.169148 15255 raft_consensus.cc:729] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45136f324a8247d2bccaa028faac1d36, State: Initialized, Role: FOLLOWER
I20260927 22:50:20.169286 15254 raft_consensus.cc:374] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:20.169379 15254 raft_consensus.cc:729] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92735147983d4ece810814e55aadc504, State: Initialized, Role: FOLLOWER
I20260927 22:50:20.169813 15255 consensus_queue.cc:260] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.169898 15254 consensus_queue.cc:260] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.170955 15105 heartbeater.cc:503] Master 127.0.97.62:45583 was elected leader, sending a full tablet report...
I20260927 22:50:20.171623 15255 ts_tablet_manager.cc:1428] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260927 22:50:20.171633 15236 heartbeater.cc:503] Master 127.0.97.62:45583 was elected leader, sending a full tablet report...
I20260927 22:50:20.172124 15254 ts_tablet_manager.cc:1428] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260927 22:50:20.179090 15260 raft_consensus.cc:482] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:50:20.179296 15260 raft_consensus.cc:504] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.180925 15260 leader_election.cc:288] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681), 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347)
I20260927 22:50:20.187083 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.181154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36681 (local address 127.0.97.3:44667)
0927 22:50:20.181584 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:20.181605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:20.181730 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:20.182278 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:20.182284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:20.182312 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:20.182511 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.182523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.183753 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.183758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.185223 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.185237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.186418 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.186425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.186597 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.186635 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:50:20.186724 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:50:20.186829 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":19}
I20260927 22:50:20.187152 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.181757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44667 (local address 127.0.97.1:36681)
0927 22:50:20.181955 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.181961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.181979 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:50:20.182079 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.182084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.182157 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.182248 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:20.182639 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.183623 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.185382 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.186260 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.186361 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.186430 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.186759 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.186904 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:50:20.187314 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.181388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42357 (local address 127.0.97.2:40347)
0927 22:50:20.181578 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.181585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.182093 (+ 508us) server_negotiation.cc:407] Connection header received
0927 22:50:20.182314 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.182318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.182386 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.182476 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:20.183040 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.183906 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.185683 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.186493 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.186607 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.186699 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.186980 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.187103 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:50:20.187507 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.181239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.97.3:42357)
0927 22:50:20.181986 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:20.182002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:20.182185 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:20.182497 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:20.182500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:20.182543 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:20.182878 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.182890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.184058 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.184061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.185546 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:20.185560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.186668 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.186672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:20.186827 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.186860 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:50:20.186948 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:50:20.187074 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":22,"thread_start_us":416,"threads_started":1}
I20260927 22:50:20.188498 15060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251d98ded52b4e68993fa68ee49785a9" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45136f324a8247d2bccaa028faac1d36" is_pre_election: true
I20260927 22:50:20.188486 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251d98ded52b4e68993fa68ee49785a9" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108029489b8f4b2f865eef7d2bb07966" is_pre_election: true
I20260927 22:50:20.188951 15060 raft_consensus.cc:2454] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 0.
I20260927 22:50:20.188951 14928 raft_consensus.cc:2454] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 0.
I20260927 22:50:20.189546 15127 leader_election.cc:302] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 108029489b8f4b2f865eef7d2bb07966, 92735147983d4ece810814e55aadc504; no voters:
I20260927 22:50:20.189898 15260 raft_consensus.cc:2781] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:50:20.190037 15260 raft_consensus.cc:482] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:50:20.190162 15260 raft_consensus.cc:3037] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:20.192497 15260 raft_consensus.cc:504] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.193050 15260 leader_election.cc:288] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 election: Requested vote from peers 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681), 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347)
I20260927 22:50:20.193482 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251d98ded52b4e68993fa68ee49785a9" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108029489b8f4b2f865eef7d2bb07966"
I20260927 22:50:20.193462 15060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251d98ded52b4e68993fa68ee49785a9" candidate_uuid: "92735147983d4ece810814e55aadc504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45136f324a8247d2bccaa028faac1d36"
I20260927 22:50:20.193655 14928 raft_consensus.cc:3037] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:20.193661 15060 raft_consensus.cc:3037] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:20.196014 14928 raft_consensus.cc:2454] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 1.
I20260927 22:50:20.196029 15060 raft_consensus.cc:2454] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92735147983d4ece810814e55aadc504 in term 1.
I20260927 22:50:20.196336 15127 leader_election.cc:302] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 108029489b8f4b2f865eef7d2bb07966, 92735147983d4ece810814e55aadc504; no voters:
I20260927 22:50:20.196570 15260 raft_consensus.cc:2781] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:20.196936 15260 raft_consensus.cc:686] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 1 LEADER]: Becoming Leader. State: Replica: 92735147983d4ece810814e55aadc504, State: Running, Role: LEADER
I20260927 22:50:20.197408 15260 consensus_queue.cc:237] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } }
I20260927 22:50:20.201484 14788 catalog_manager.cc:5495] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92735147983d4ece810814e55aadc504 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "92735147983d4ece810814e55aadc504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92735147983d4ece810814e55aadc504" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41759 } health_report { overall_health: HEALTHY } } }
W20260927 22:50:20.202395 14974 tablet.cc:2369] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:20.220752 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.215739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.0.97.3:41759)
0927 22:50:20.215908 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.215912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.215978 (+ 66us) server_negotiation.cc:407] Connection header received
0927 22:50:20.216154 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.216157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.216228 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.216336 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:50:20.216746 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.217436 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.218770 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.219150 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.219235 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.219740 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:50:20.220255 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:50:20.220357 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.220500 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.220564 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:50:20.228781 14928 raft_consensus.cc:1261] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Refusing update from remote peer 92735147983d4ece810814e55aadc504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:20.228773 15060 raft_consensus.cc:1261] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Refusing update from remote peer 92735147983d4ece810814e55aadc504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:20.229638 15262 consensus_queue.cc:1035] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45136f324a8247d2bccaa028faac1d36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:20.229969 15260 consensus_queue.cc:1035] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "108029489b8f4b2f865eef7d2bb07966" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:20.250715 14788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:20.250983 14788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:20.254612 14788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Sending DeleteTablet for 3 replicas of tablet 251d98ded52b4e68993fa68ee49785a9
I20260927 22:50:20.255689 14908 tablet_service.cc:1514] Processing DeleteTablet for tablet 251d98ded52b4e68993fa68ee49785a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58416
I20260927 22:50:20.255914 15040 tablet_service.cc:1514] Processing DeleteTablet for tablet 251d98ded52b4e68993fa68ee49785a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54280
I20260927 22:50:20.256054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14825
I20260927 22:50:20.256152 15171 tablet_service.cc:1514] Processing DeleteTablet for tablet 251d98ded52b4e68993fa68ee49785a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38398
I20260927 22:50:20.268457 15273 tablet_replica.cc:330] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: stopping tablet replica
I20260927 22:50:20.268958 15273 raft_consensus.cc:2229] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:20.269523 15273 raft_consensus.cc:2258] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:20.270363 15274 tablet_replica.cc:330] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: stopping tablet replica
I20260927 22:50:20.270771 15274 raft_consensus.cc:2229] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:20.271265 15274 raft_consensus.cc:2258] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:20.272094 15275 tablet_replica.cc:330] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: stopping tablet replica
I20260927 22:50:20.272480 15275 raft_consensus.cc:2229] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:50:20.273118 15275 raft_consensus.cc:2258] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:20.273623 15273 ts_tablet_manager.cc:1905] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:20.275187 15274 ts_tablet_manager.cc:1905] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:20.277465 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.256871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.0.97.1:36681)
0927 22:50:20.257437 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.257443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.257460 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:50:20.257513 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.257518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.257622 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.257720 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:20.259744 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.260985 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.263281 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.264032 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.264161 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.268209 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:20.268228 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:20.268251 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:20.268292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:20.273201 (+ 4909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.274342 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.274351 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.274355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.274510 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.275092 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.275097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.275101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.275740 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:20.276020 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.276822 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.277057 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439}
I20260927 22:50:20.279253 14968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:20.282891 15275 ts_tablet_manager.cc:1905] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:20.289901 15274 ts_tablet_manager.cc:1918] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:20.290174 15274 log.cc:1196] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/251d98ded52b4e68993fa68ee49785a9
I20260927 22:50:20.290793 15275 ts_tablet_manager.cc:1918] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:20.290973 15275 log.cc:1196] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/251d98ded52b4e68993fa68ee49785a9
I20260927 22:50:20.291023 15274 ts_tablet_manager.cc:1939] T 251d98ded52b4e68993fa68ee49785a9 P 45136f324a8247d2bccaa028faac1d36: Deleting consensus metadata
I20260927 22:50:20.291580 15275 ts_tablet_manager.cc:1939] T 251d98ded52b4e68993fa68ee49785a9 P 92735147983d4ece810814e55aadc504: Deleting consensus metadata
I20260927 22:50:20.293426 14772 catalog_manager.cc:4841] TS 45136f324a8247d2bccaa028faac1d36 (127.0.97.2:40347): tablet 251d98ded52b4e68993fa68ee49785a9 (table test-table [id=ef971cb341e64987a64ed2221e7e2f11]) successfully deleted
I20260927 22:50:20.293783 14772 catalog_manager.cc:4841] TS 92735147983d4ece810814e55aadc504 (127.0.97.3:41759): tablet 251d98ded52b4e68993fa68ee49785a9 (table test-table [id=ef971cb341e64987a64ed2221e7e2f11]) successfully deleted
I20260927 22:50:20.376693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14825
I20260927 22:50:20.377938 15273 ts_tablet_manager.cc:1918] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:20.378122 15273 log.cc:1196] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/251d98ded52b4e68993fa68ee49785a9
I20260927 22:50:20.378561 15273 ts_tablet_manager.cc:1939] T 251d98ded52b4e68993fa68ee49785a9 P 108029489b8f4b2f865eef7d2bb07966: Deleting consensus metadata
I20260927 22:50:20.380205 14775 catalog_manager.cc:4841] TS 108029489b8f4b2f865eef7d2bb07966 (127.0.97.1:36681): tablet 251d98ded52b4e68993fa68ee49785a9 (table test-table [id=ef971cb341e64987a64ed2221e7e2f11]) successfully deleted
I20260927 22:50:20.403019 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14977
I20260927 22:50:20.416767 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.403873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.0.97.2:40347)
0927 22:50:20.404054 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.404059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.404078 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:50:20.404193 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.404199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.404279 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.404359 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:20.405597 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.406338 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.407649 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.408071 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.408180 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.410806 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:20.410820 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:20.410836 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:20.410878 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:20.413595 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.414269 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.414277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.414280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.414409 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.414828 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.414834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.414838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.415356 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:20.415936 (+ 580us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.416166 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.416397 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:50:20.417521 15100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:20.504319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14977
I20260927 22:50:20.518880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 15108
I20260927 22:50:20.534404 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:20.519705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.0.97.3:41759)
0927 22:50:20.519879 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:20.519884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:20.519990 (+ 106us) server_negotiation.cc:407] Connection header received
0927 22:50:20.520146 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:20.520150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:20.520212 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:20.520309 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:20.522092 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.522899 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.524768 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:20.525188 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:20.525285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:20.528697 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:20.528709 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:20.528723 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:20.528769 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:20.531640 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.532425 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.532434 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.532436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.532575 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:20.532986 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:20.532989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:20.532991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:20.533326 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:20.533552 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:20.533919 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 22:50:20.534076 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:50:20.535161 15231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:20.652797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 15108
I20260927 22:50:20.670769 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 14755
I20260927 22:50:20.672119 14818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:20.780053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 14755
I20260927 22:50:20.798010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--webserver_interface=127.0.97.62
--webserver_port=44449
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:21.110996 15280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:21.111279 15280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:21.111406 15280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:21.118594 15280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:21.118690 15280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:21.118733 15280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:21.118788 15280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:21.118829 15280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:21.128522 15280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:21.129743 15280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:21.130959 15280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:21.142207 15285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:21.142246 15288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:21.142177 15286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:21.143378 15280 server_base.cc:1034] running on GCE node
I20260927 22:50:21.144238 15280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:21.145721 15280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:21.146941 15280 hybrid_clock.cc:648] HybridClock initialized: now 1790549421146874 us; error 62 us; skew 500 ppm
I20260927 22:50:21.147274 15280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:21 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:21.154042 15280 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:50:21.155650 15280 webserver.cc:460] Webserver started at http://127.0.97.62:44449/ using document root <none> and password file <none>
I20260927 22:50:21.156333 15280 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:21.156446 15280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:21.161939 15280 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:50:21.164923 15295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:21.166306 15280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:50:21.166497 15280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "5d12937fe9554537b99eec1167aaa3f3"
format_stamp: "Formatted at 2026-09-27 22:50:15 on dist-test-slave-kr25"
server_key: "597fc14b1da3b55888fee6ab165588f1"
server_key_iv: "57ca5cc0066cdf921f7d104d8fec184d"
server_key_version: "encryptionkey@0"
I20260927 22:50:21.167526 15280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:21.179873 15280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:21.183871 15280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:21.184127 15280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:21.201355 15280 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45583
I20260927 22:50:21.201409 15346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45583 every 8 connection(s)
I20260927 22:50:21.202786 15280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:50:21.207370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 15280
I20260927 22:50:21.208343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36681
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45583
--webserver_port=38399
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:50:21.210160 15347 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:50:21.214418 15347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Bootstrap starting.
I20260927 22:50:21.234628 15347 log.cc:824] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:21.250034 15347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:50:21.250521 15347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3: Bootstrap complete.
I20260927 22:50:21.259204 15347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:21.260113 15347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d12937fe9554537b99eec1167aaa3f3, State: Initialized, Role: FOLLOWER
I20260927 22:50:21.260699 15347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:21.260911 15347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:50:21.261019 15347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:50:21.261179 15347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:50:21.265112 15347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:21.265626 15347 leader_election.cc:302] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d12937fe9554537b99eec1167aaa3f3; no voters:
I20260927 22:50:21.266242 15347 leader_election.cc:288] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:50:21.266394 15351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:50:21.267391 15351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [term 3 LEADER]: Becoming Leader. State: Replica: 5d12937fe9554537b99eec1167aaa3f3, State: Running, Role: LEADER
I20260927 22:50:21.267993 15351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260927 22:50:21.268318 15347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:21.270381 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5d12937fe9554537b99eec1167aaa3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260927 22:50:21.270758 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:21.273021 15353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d12937fe9554537b99eec1167aaa3f3. Latest consensus state: current_term: 3 leader_uuid: "5d12937fe9554537b99eec1167aaa3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12937fe9554537b99eec1167aaa3f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260927 22:50:21.273293 15353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12937fe9554537b99eec1167aaa3f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:21.274413 15356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:50:21.282454 15356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:50:21.283892 15356 catalog_manager.cc:1259] Loaded cluster ID: 7c61a41934574a48a367ca60d68b273a
I20260927 22:50:21.283984 15356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 22:50:21.285853 15356 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0a8f4759ad google::LogMessage::Fail() at ??:0
@ 0x7f0a8f47888a google::LogMessage::SendToLog() at ??:0
@ 0x7f0a8f4754de google::LogMessage::Flush() at ??:0
@ 0x7f0a8f476a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0aa4ea8512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0aa4ea5976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0aa4f2d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0aa4f2d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0a920c1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0a920c491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0a920c4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0a9209feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0a953796db start_thread at ??:0
@ 0x7f0a8e30f61f clone at ??:0
W20260927 22:50:21.610190 15349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:21.610461 15349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:21.610554 15349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:21.617717 15349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:21.617833 15349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:21.617980 15349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:50:21.627673 15349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38399
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:21.628955 15349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:21.630214 15349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:21.642647 15388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:21.642627 15389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:21.642596 15391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:21.643364 15349 server_base.cc:1034] running on GCE node
I20260927 22:50:21.643972 15349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:21.644857 15349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:21.646060 15349 hybrid_clock.cc:648] HybridClock initialized: now 1790549421646008 us; error 41 us; skew 500 ppm
I20260927 22:50:21.646358 15349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:21 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:21.649472 15349 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:50:21.650868 15349 webserver.cc:460] Webserver started at http://127.0.97.1:38399/ using document root <none> and password file <none>
I20260927 22:50:21.651563 15349 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:21.651692 15349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:21.657048 15349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:50:21.659631 15398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:21.660993 15349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:50:21.661146 15349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "108029489b8f4b2f865eef7d2bb07966"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "598fb0845a1c2d69fb0972ddb6668573"
server_key_iv: "af6f6c99bf02bd427c7d1b06fab361e8"
server_key_version: "encryptionkey@0"
I20260927 22:50:21.662123 15349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:21.683763 15349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:21.687850 15349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:21.688103 15349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:21.689054 15349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:21.690368 15349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:21.690464 15349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:21.690573 15349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:21.690641 15349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:21.720834 15349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36681
I20260927 22:50:21.720901 15510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36681 every 8 connection(s)
I20260927 22:50:21.722226 15349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:50:21.722982 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 15349
I20260927 22:50:21.723695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40347
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45583
--webserver_port=46337
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:50:21.727496 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:21.725611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:41489)
0927 22:50:21.726368 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:21.726560 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":304,"threads_started":1}
W20260927 22:50:21.728034 15511 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45583 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45583: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
I20260927 22:50:21.728781 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:21.728497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:34435)
0927 22:50:21.728623 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:21.728686 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:50:21.729627 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:21.729242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:58485)
0927 22:50:21.729429 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:21.729509 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 22:50:21.729851 15511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:50:22.033820 15514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:22.034126 15514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:22.034265 15514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:22.041548 15514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:22.041675 15514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:22.041831 15514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:50:22.051406 15514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46337
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:22.052678 15514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:22.053956 15514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:22.066556 15519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:22.066568 15522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:22.066607 15520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:22.067390 15514 server_base.cc:1034] running on GCE node
I20260927 22:50:22.068053 15514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:22.069218 15514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:22.070441 15514 hybrid_clock.cc:648] HybridClock initialized: now 1790549422070367 us; error 59 us; skew 500 ppm
I20260927 22:50:22.070812 15514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:22 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:22.073966 15514 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:50:22.075348 15514 webserver.cc:460] Webserver started at http://127.0.97.2:46337/ using document root <none> and password file <none>
I20260927 22:50:22.076027 15514 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:22.076143 15514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:22.081399 15514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:50:22.086074 15529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:22.087339 15514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:50:22.087540 15514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "45136f324a8247d2bccaa028faac1d36"
format_stamp: "Formatted at 2026-09-27 22:50:16 on dist-test-slave-kr25"
server_key: "6b2dd35ce6d707e3f3ebbf9f9b1ea743"
server_key_iv: "d28d4631bf847063351f60cf367389f7"
server_key_version: "encryptionkey@0"
I20260927 22:50:22.088536 15514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:22.108960 15514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:22.113237 15514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:22.113493 15514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:22.114488 15514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:22.115818 15514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:22.115912 15514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:22.116017 15514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:22.116082 15514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:22.143332 15514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40347
I20260927 22:50:22.143393 15641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40347 every 8 connection(s)
I20260927 22:50:22.144790 15514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:50:22.149904 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:22.147939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:45067)
0927 22:50:22.148784 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:22.148997 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
W20260927 22:50:22.150545 15642 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45583 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45583: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
I20260927 22:50:22.151336 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:22.151024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:50161)
0927 22:50:22.151166 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:22.151234 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:50:22.152243 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:22.151951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:34111)
0927 22:50:22.152073 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:22.152145 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:50:22.152464 15642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:50:22.154891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 15514
I20260927 22:50:22.155578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41759
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45583
--webserver_port=39305
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:22.463963 15645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:22.464272 15645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:22.464427 15645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:22.471901 15645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:22.472013 15645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:22.472165 15645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:50:22.482302 15645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39305
--tserver_master_addrs=127.0.97.62:45583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:22.483780 15645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:22.485062 15645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:22.497181 15651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:22.497331 15650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:22.497213 15653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:22.497925 15645 server_base.cc:1034] running on GCE node
I20260927 22:50:22.498524 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:22.499395 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:22.500648 15645 hybrid_clock.cc:648] HybridClock initialized: now 1790549422500593 us; error 39 us; skew 500 ppm
I20260927 22:50:22.500967 15645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:22 dist-test-slave-kr25 krb5kdc[14205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:22.504176 15645 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:50:22.505787 15645 webserver.cc:460] Webserver started at http://127.0.97.3:39305/ using document root <none> and password file <none>
I20260927 22:50:22.506481 15645 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:22.506600 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:22.512188 15645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:50:22.514955 15660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:22.516291 15645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:50:22.516477 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "92735147983d4ece810814e55aadc504"
format_stamp: "Formatted at 2026-09-27 22:50:17 on dist-test-slave-kr25"
server_key: "9f89fd527e8ff006a2b8d6902a3ff90c"
server_key_iv: "9ccff55aabaa07d74971abee81e37757"
server_key_version: "encryptionkey@0"
I20260927 22:50:22.517406 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:22.534441 15645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:22.538256 15645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:22.538506 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:22.539501 15645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:22.540786 15645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:22.540884 15645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:22.540997 15645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:22.541069 15645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:22.571975 15645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41759
I20260927 22:50:22.572026 15772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41759 every 8 connection(s)
I20260927 22:50:22.573415 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:50:22.575807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 15645
I20260927 22:50:22.578265 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:22.576254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:42861)
0927 22:50:22.577126 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:22.577364 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":325,"threads_started":1}
W20260927 22:50:22.578970 15773 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45583 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45583: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
I20260927 22:50:22.579771 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:22.579480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:42843)
0927 22:50:22.579608 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:22.579676 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 22:50:22.580610 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:22.580277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:57907)
0927 22:50:22.580416 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:22.580497 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:50:22.580848 15773 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:50:22.731048 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:22.730323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:60953)
0927 22:50:22.730732 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:22.730817 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":193,"threads_started":1}
I20260927 22:50:23.154018 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:23.153059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:35023)
0927 22:50:23.153612 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:23.153721 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260927 22:50:23.582131 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:23.581370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:52145)
0927 22:50:23.581798 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:23.581882 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":198,"threads_started":1}
I20260927 22:50:23.732640 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:23.731882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:57889)
0927 22:50:23.732400 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:23.732476 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260927 22:50:24.155650 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:24.154879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:49065)
0927 22:50:24.155325 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:24.155477 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260927 22:50:24.583699 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:24.582969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:46293)
0927 22:50:24.583409 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:24.583534 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":210,"threads_started":1}
I20260927 22:50:24.734038 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:24.733381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:56267)
0927 22:50:24.733793 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:24.733868 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260927 22:50:25.157262 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:25.156498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:42629)
0927 22:50:25.156977 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:25.157076 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":216,"threads_started":1}
I20260927 22:50:25.585176 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:25.584454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:39937)
0927 22:50:25.584918 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:25.585002 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260927 22:50:25.735925 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:25.734872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:44519)
0927 22:50:25.735630 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:25.735709 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":566,"threads_started":1}
I20260927 22:50:26.158838 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:26.158088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:33365)
0927 22:50:26.158551 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:26.158634 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260927 22:50:26.586865 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:26.586158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:33471)
0927 22:50:26.586576 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:26.586662 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":189,"threads_started":1}
I20260927 22:50:26.737488 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:26.736808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:53257)
0927 22:50:26.737242 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:26.737320 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":221,"threads_started":1}
I20260927 22:50:27.160712 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:27.159932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:39923)
0927 22:50:27.160444 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:27.160542 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":261,"threads_started":1}
I20260927 22:50:27.588521 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:27.587789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:48923)
0927 22:50:27.588260 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:27.588342 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260927 22:50:27.738981 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:27.738252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:40361)
0927 22:50:27.738720 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:27.738826 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":269,"threads_started":1}
I20260927 22:50:28.162492 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:28.161717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:54527)
0927 22:50:28.162200 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:28.162316 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":247,"threads_started":1}
I20260927 22:50:28.590015 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:28.589319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:43467)
0927 22:50:28.589772 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:28.589856 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":256,"threads_started":1}
I20260927 22:50:28.740585 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:28.739855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:41509)
0927 22:50:28.740297 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:28.740375 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":188,"threads_started":1}
I20260927 22:50:29.164300 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:29.163412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:35913)
0927 22:50:29.164009 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:29.164090 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":227,"threads_started":1}
I20260927 22:50:29.591804 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:29.590965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:51701)
0927 22:50:29.591497 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:29.591584 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":214,"threads_started":1}
I20260927 22:50:29.741986 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:29.741363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:51309)
0927 22:50:29.741757 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:29.741831 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
I20260927 22:50:30.165983 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:30.165227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:41495)
0927 22:50:30.165734 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:30.165821 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":297,"threads_started":1}
I20260927 22:50:30.593405 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:30.592706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:35953)
0927 22:50:30.593164 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:30.593246 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260927 22:50:30.743415 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:30.742731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:43567)
0927 22:50:30.743163 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:30.743247 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":211,"threads_started":1}
I20260927 22:50:31.167698 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:31.166877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:48765)
0927 22:50:31.167357 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:31.167517 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":230,"threads_started":1}
I20260927 22:50:31.594969 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:31.594247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:33877)
0927 22:50:31.594722 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:31.594799 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":218,"threads_started":1}
I20260927 22:50:31.745023 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:31.744337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:38229)
0927 22:50:31.744775 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:31.744858 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":239,"threads_started":1}
I20260927 22:50:32.169340 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:32.168637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:36539)
0927 22:50:32.169094 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:32.169178 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
I20260927 22:50:32.596518 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:32.595852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:53431)
0927 22:50:32.596280 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:32.596359 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":239,"threads_started":1}
I20260927 22:50:32.746762 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:32.745894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:45077)
0927 22:50:32.746416 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:32.746528 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":305,"threads_started":1}
I20260927 22:50:33.171044 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:33.170313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:35981)
0927 22:50:33.170774 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:33.170865 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260927 22:50:33.598223 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:33.597370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:54119)
0927 22:50:33.597861 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:33.597997 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
I20260927 22:50:33.748288 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:33.747631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:38573)
0927 22:50:33.748038 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:33.748132 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":201,"threads_started":1}
I20260927 22:50:34.172588 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:34.171895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:43951)
0927 22:50:34.172325 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:34.172411 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":222,"threads_started":1}
I20260927 22:50:34.599865 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:34.599157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:55621)
0927 22:50:34.599623 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:34.599709 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":278,"threads_started":1}
I20260927 22:50:34.749810 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:34.749109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:42263)
0927 22:50:34.749532 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:34.749617 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":200,"threads_started":1}
I20260927 22:50:35.174397 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:35.173605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:54661)
0927 22:50:35.174117 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:35.174208 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":259,"threads_started":1}
I20260927 22:50:35.601464 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:35.600713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:40235)
0927 22:50:35.601184 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:35.601266 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":234,"threads_started":1}
I20260927 22:50:35.751482 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:35.750688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:33249)
0927 22:50:35.751164 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:35.751251 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260927 22:50:36.176000 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:36.175273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:48795)
0927 22:50:36.175741 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:36.175821 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
I20260927 22:50:36.602893 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:36.602228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:53789)
0927 22:50:36.602640 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:36.602716 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":200,"threads_started":1}
I20260927 22:50:36.753273 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:36.752556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.1:45545)
0927 22:50:36.753017 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:36.753097 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260927 22:50:37.177695 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:37.176999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.2:43571)
0927 22:50:37.177434 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:37.177516 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":214,"threads_started":1}
I20260927 22:50:37.580847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 15349
I20260927 22:50:37.598448 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:37.581918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.0.97.1:36681)
0927 22:50:37.582503 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:37.582522 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:37.582566 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:50:37.584673 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:37.584683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:37.585100 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:37.585388 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:37.586650 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:37.587898 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:37.589044 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:37.589698 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:37.589831 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:37.592482 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:37.592490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:37.592508 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:37.592549 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:37.595537 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:37.596189 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:37.596195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:37.596202 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:37.596362 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:37.596757 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:37.596760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:37.596762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:37.597273 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:37.597484 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:37.597848 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 22:50:37.598180 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
I20260927 22:50:37.599838 15506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:37.604563 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:37.603871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.97.3:37781)
0927 22:50:37.604310 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:37.604385 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260927 22:50:37.682286 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 15349
I20260927 22:50:37.696163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 15514
I20260927 22:50:37.712651 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:37.696976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.0.97.2:40347)
0927 22:50:37.697453 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:37.697467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:37.697502 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:50:37.698949 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:37.698959 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:37.699394 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:37.699735 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:37.701012 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:37.702186 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:37.703386 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:37.704100 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:37.704250 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:37.706753 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:37.706762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:37.706777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:37.706812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:37.709712 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:37.710332 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:37.710338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:37.710344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:37.710502 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:37.710804 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:37.710807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:37.710809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:37.711311 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:37.711577 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:37.712013 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 22:50:37.712351 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260927 22:50:37.714068 15637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:37.796613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 15514
I20260927 22:50:37.810093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 15645
I20260927 22:50:37.832015 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:37.810906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.0.97.3:41759)
0927 22:50:37.811425 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:37.811462 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:37.811508 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:50:37.813111 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:37.813119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:37.813541 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:37.813849 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:37.815157 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:37.816419 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:37.817667 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:37.818316 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:37.818478 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:37.820874 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:37.820882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:37.820897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:37.820932 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:37.828745 (+ 7813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:37.829404 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:37.829411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:37.829418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:37.829578 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:37.829931 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:37.829935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:37.829938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:37.830472 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:37.830741 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:37.831201 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 22:50:37.831702 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":218,"threads_started":1}
I20260927 22:50:37.833389 15768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:37.913928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 15645
2026-09-27T22:50:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23871 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:50:38 dist-test-slave-kr25 krb5kdc[15830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:50:38 dist-test-slave-kr25 krb5kdc[15830](info): set up 2 sockets
Sep 27 22:50:38 dist-test-slave-kr25 krb5kdc[15830](info): commencing operation
krb5kdc: starting...
W20260927 22:50:38.919685 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:50:39 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:50:39.138489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34779
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:39.449570 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:39.449870 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:39.449985 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:39.457160 15843 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:50:39.457250 15843 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:50:39.457309 15843 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:50:39.457377 15843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:50:39.457446 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:39.457540 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:39.457604 15843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:50:39.457679 15843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:50:39.467418 15843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:39.468672 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:39.469951 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:39.481302 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:39.481357 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:39.481359 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:39.482020 15843 server_base.cc:1034] running on GCE node
I20260927 22:50:39.482890 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:39.484463 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:39.485704 15843 hybrid_clock.cc:648] HybridClock initialized: now 1790549439485635 us; error 41 us; skew 500 ppm
I20260927 22:50:39.486040 15843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:39 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:39.489197 15843 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:50:39.490659 15843 webserver.cc:460] Webserver started at http://127.0.97.62:42911/ using document root <none> and password file <none>
I20260927 22:50:39.491345 15843 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:39.491487 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:39.491801 15843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:39.494777 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "24f8d4caeee54e89940b5fffc77edf0c"
format_stamp: "Formatted at 2026-09-27 22:50:39 on dist-test-slave-kr25"
server_key: "32bead642361ba73cb8a8534c1e95d24"
server_key_iv: "55fca012781e3f4b0c9e7584d246405b"
server_key_version: "encryptionkey@0"
I20260927 22:50:39.495607 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "24f8d4caeee54e89940b5fffc77edf0c"
format_stamp: "Formatted at 2026-09-27 22:50:39 on dist-test-slave-kr25"
server_key: "32bead642361ba73cb8a8534c1e95d24"
server_key_iv: "55fca012781e3f4b0c9e7584d246405b"
server_key_version: "encryptionkey@0"
I20260927 22:50:39.500728 15843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:50:39.504200 15858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:39.505496 15843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:50:39.505674 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "24f8d4caeee54e89940b5fffc77edf0c"
format_stamp: "Formatted at 2026-09-27 22:50:39 on dist-test-slave-kr25"
server_key: "32bead642361ba73cb8a8534c1e95d24"
server_key_iv: "55fca012781e3f4b0c9e7584d246405b"
server_key_version: "encryptionkey@0"
I20260927 22:50:39.505863 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:39.525166 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:39.529246 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:39.529485 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:39.545755 15843 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34779
I20260927 22:50:39.545774 15909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34779 every 8 connection(s)
I20260927 22:50:39.547070 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:50:39.548660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 15843
I20260927 22:50:39.548877 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:50:39.549719 388 external_mini_cluster.cc:1428] Setting key 1894874e094b9059e1a0af1eebc3770e
I20260927 22:50:39.551627 15910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:50:39 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:39.564951 15910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Bootstrap starting.
I20260927 22:50:39.568346 15910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:39.569562 15910 log.cc:824] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:39.572696 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:39.551692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.0.97.62:34779)
0927 22:50:39.552455 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:39.552467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:39.552507 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:50:39.554162 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:39.554186 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:39.554597 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:39.554951 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:39.556715 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:39.558016 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:39.559860 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:39.560510 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:39.560646 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:39.565275 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:39.565283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:39.565297 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:39.565333 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:39.568429 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:39.569184 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:39.569190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:39.569195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:39.569356 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:39.569759 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:39.569764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:39.569768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:39.570288 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:39.570516 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:39.571031 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 22:50:39.571421 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":354,"threads_started":1}
I20260927 22:50:39.572809 15910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: No bootstrap required, opened a new log
I20260927 22:50:39.580143 15910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:50:39.580446 15910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:39.580525 15910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f8d4caeee54e89940b5fffc77edf0c, State: Initialized, Role: FOLLOWER
I20260927 22:50:39.581027 15910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:50:39.581277 15910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:50:39.581408 15910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:50:39.581597 15910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:39.583598 15910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:50:39.584034 15910 leader_election.cc:302] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f8d4caeee54e89940b5fffc77edf0c; no voters:
I20260927 22:50:39.584595 15910 leader_election.cc:288] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:50:39.584726 15915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:39.585664 15915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 LEADER]: Becoming Leader. State: Replica: 24f8d4caeee54e89940b5fffc77edf0c, State: Running, Role: LEADER
I20260927 22:50:39.586149 15915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:50:39.586549 15910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:50:39.588475 15917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f8d4caeee54e89940b5fffc77edf0c. Latest consensus state: current_term: 1 leader_uuid: "24f8d4caeee54e89940b5fffc77edf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } } }
I20260927 22:50:39.588816 15917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:39.589309 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24f8d4caeee54e89940b5fffc77edf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } } }
I20260927 22:50:39.589610 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:50:39.590883 15922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:50:39.596452 15922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:50:39.604875 15922 catalog_manager.cc:1347] Generated new cluster ID: 0bea33be67514e8798d611dea82b50df
I20260927 22:50:39.604983 15922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:50:39.624246 15922 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:50:39.625516 15922 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:50:39.633149 15922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Generated new TSK 0
I20260927 22:50:39.634064 15922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:50:39.702265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34779
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:40.014580 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:40.014843 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:40.014979 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:40.022001 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:40.022099 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:40.022244 15937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:50:40.031752 15937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:40.033051 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:40.034396 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:40.046875 15945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:40.046918 15943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:40.047160 15942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:40.047703 15937 server_base.cc:1034] running on GCE node
I20260927 22:50:40.048260 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:40.049165 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:40.050350 15937 hybrid_clock.cc:648] HybridClock initialized: now 1790549440050297 us; error 41 us; skew 500 ppm
I20260927 22:50:40.050657 15937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:40 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:40.053620 15937 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:50:40.054884 15937 webserver.cc:460] Webserver started at http://127.0.97.1:41801/ using document root <none> and password file <none>
I20260927 22:50:40.055552 15937 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:40.055655 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:40.055896 15937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:40.058534 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "db67e52227504cefa263cb9ed1987f97"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "be7a5aa00173983e0a25622f27ef8324"
server_key_iv: "340a54aae8ac335579f81d995f774054"
server_key_version: "encryptionkey@0"
I20260927 22:50:40.059201 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "db67e52227504cefa263cb9ed1987f97"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "be7a5aa00173983e0a25622f27ef8324"
server_key_iv: "340a54aae8ac335579f81d995f774054"
server_key_version: "encryptionkey@0"
I20260927 22:50:40.063959 15937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:50:40.067153 15952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:40.068450 15937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:50:40.068578 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "db67e52227504cefa263cb9ed1987f97"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "be7a5aa00173983e0a25622f27ef8324"
server_key_iv: "340a54aae8ac335579f81d995f774054"
server_key_version: "encryptionkey@0"
I20260927 22:50:40.068740 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:40.080799 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:40.084844 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:40.085074 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:40.085933 15937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:40.087200 15937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:40.087277 15937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:40.087390 15937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:40.087472 15937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:40.116248 15937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39491
I20260927 22:50:40.116293 16064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39491 every 8 connection(s)
I20260927 22:50:40.117473 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:50:40.124785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 15937
I20260927 22:50:40.124963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:50:40.125555 388 external_mini_cluster.cc:1428] Setting key 9450708a2b59b214200f48050dc5a90e
Sep 27 22:50:40 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:40.136946 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:40.119811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41595 (local address 127.0.97.62:34779)
0927 22:50:40.120229 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:40.120233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:40.121223 (+ 990us) server_negotiation.cc:407] Connection header received
0927 22:50:40.123013 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:40.123019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:40.123094 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:40.123187 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:40.124765 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.125537 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.127275 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.127861 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.128004 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:40.131543 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:40.131549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:40.131553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:40.131582 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:40.134064 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:40.134706 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:40.134712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:40.134716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:40.134831 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:40.135257 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:40.135272 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:40.135275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:40.135674 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:40.135812 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:40.136396 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 22:50:40.136597 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
I20260927 22:50:40.137782 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:40.120390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:41595)
0927 22:50:40.121063 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:40.121112 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:40.122758 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:40.123421 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:40.123428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:40.123960 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:40.124591 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:40.124616 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.125710 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.125716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:40.127151 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:40.127166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.128163 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.128175 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:40.128386 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:40.129040 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:40.129069 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:40.131318 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:40.134186 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:40.134194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:40.134209 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:40.134601 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:40.134927 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:40.134932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:40.134935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:40.135112 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:40.135782 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:40.135788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:40.136081 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 22:50:40.136295 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:50:40.136668 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":264,"threads_started":1}
I20260927 22:50:40.139145 16065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34779
I20260927 22:50:40.139467 16065 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:40.140265 16065 heartbeater.cc:511] Master 127.0.97.62:34779 requested a full tablet report, sending...
I20260927 22:50:40.142210 15875 ts_manager.cc:194] Registered new tserver with Master: db67e52227504cefa263cb9ed1987f97 (127.0.97.1:39491)
I20260927 22:50:40.143610 15875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41595
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:50:40.175069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34779
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:40.485034 16071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:40.485328 16071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:40.485450 16071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:40.492878 16071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:40.492995 16071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:40.493158 16071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:50:40.503230 16071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:40.504511 16071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:40.505757 16071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:40.518718 16076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:40.518776 16077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:40.518747 16079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:40.519570 16071 server_base.cc:1034] running on GCE node
I20260927 22:50:40.520169 16071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:40.521119 16071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:40.522316 16071 hybrid_clock.cc:648] HybridClock initialized: now 1790549440522232 us; error 72 us; skew 500 ppm
I20260927 22:50:40.522682 16071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:40 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:40.525870 16071 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:50:40.527312 16071 webserver.cc:460] Webserver started at http://127.0.97.2:34275/ using document root <none> and password file <none>
I20260927 22:50:40.528007 16071 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:40.528105 16071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:40.528358 16071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:40.531308 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "f4585d7a5c5048cea194b42a1de1c5ab"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "b3166e8f2bb99ffe773d322a921564a0"
server_key_iv: "080d9168a44f02cbcf8adf1aecce729b"
server_key_version: "encryptionkey@0"
I20260927 22:50:40.532066 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f4585d7a5c5048cea194b42a1de1c5ab"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "b3166e8f2bb99ffe773d322a921564a0"
server_key_iv: "080d9168a44f02cbcf8adf1aecce729b"
server_key_version: "encryptionkey@0"
I20260927 22:50:40.537026 16071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:50:40.540267 16086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:40.541520 16071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:50:40.541656 16071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "f4585d7a5c5048cea194b42a1de1c5ab"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "b3166e8f2bb99ffe773d322a921564a0"
server_key_iv: "080d9168a44f02cbcf8adf1aecce729b"
server_key_version: "encryptionkey@0"
I20260927 22:50:40.541826 16071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:40.557348 16071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:40.561399 16071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:40.561650 16071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:40.562601 16071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:40.563910 16071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:40.563978 16071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:40.564090 16071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:40.564139 16071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:40.591547 16071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40019
I20260927 22:50:40.591593 16198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40019 every 8 connection(s)
I20260927 22:50:40.592902 16071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:50:40.595153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16071
I20260927 22:50:40.595319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:50:40.595834 388 external_mini_cluster.cc:1428] Setting key 993c44a50193b5d45d171800b83f4e8a
Sep 27 22:50:40 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:40.612488 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:40.595407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34583 (local address 127.0.97.62:34779)
0927 22:50:40.595638 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:40.595644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:40.596851 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 22:50:40.598918 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:40.598923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:40.599002 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:40.599084 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:40.600572 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.601304 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.602793 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.603198 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.603306 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:40.606826 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:40.606833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:40.606838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:40.606875 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:40.609682 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:40.610373 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:40.610377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:40.610379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:40.610503 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:40.610925 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:40.610928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:40.610930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:40.611170 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:40.611273 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:40.612019 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 22:50:40.612249 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:50:40.613783 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:40.595950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:34583)
0927 22:50:40.596688 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:40.596744 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:40.598667 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:40.599322 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:40.599330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:40.599858 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:40.600412 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:40.600428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.601448 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.601452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:40.602658 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:40.602671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:40.603402 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:40.603408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:40.603648 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:40.604296 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:40.604324 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:40.606632 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:40.609801 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:40.609807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:40.609819 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:40.610219 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:40.610605 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:40.610610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:40.610613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:40.610834 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:40.611272 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:40.611279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:40.611684 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 22:50:40.611943 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 22:50:40.612472 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":295,"threads_started":1}
I20260927 22:50:40.615219 16199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34779
I20260927 22:50:40.615537 16199 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:40.616209 16199 heartbeater.cc:511] Master 127.0.97.62:34779 requested a full tablet report, sending...
I20260927 22:50:40.617573 15875 ts_manager.cc:194] Registered new tserver with Master: f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:50:40.618359 15875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:50:40.644747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34779
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:50:40.960609 16204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:50:40.960909 16204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:50:40.961038 16204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:50:40.968103 16204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:50:40.968214 16204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:50:40.968366 16204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:50:40.977844 16204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:50:40.979032 16204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:50:40.980281 16204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:50:40.992388 16210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:40.992426 16209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:50:40.992460 16212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:50:40.993341 16204 server_base.cc:1034] running on GCE node
I20260927 22:50:40.993960 16204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:50:40.994877 16204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:50:40.996098 16204 hybrid_clock.cc:648] HybridClock initialized: now 1790549440996042 us; error 42 us; skew 500 ppm
I20260927 22:50:40.996397 16204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:50:40 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:50:40.999310 16204 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:50:41.000675 16204 webserver.cc:460] Webserver started at http://127.0.97.3:43265/ using document root <none> and password file <none>
I20260927 22:50:41.001343 16204 fs_manager.cc:359] Metadata directory not provided
I20260927 22:50:41.001464 16204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:50:41.001746 16204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:50:41.004626 16204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "555105c180d9494f925fca4824d05d0a"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "742f4fc6a640bb1b0040d2649acbb3b7"
server_key_iv: "497d730120cc5998859af570ff80524d"
server_key_version: "encryptionkey@0"
I20260927 22:50:41.005326 16204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "555105c180d9494f925fca4824d05d0a"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "742f4fc6a640bb1b0040d2649acbb3b7"
server_key_iv: "497d730120cc5998859af570ff80524d"
server_key_version: "encryptionkey@0"
I20260927 22:50:41.010164 16204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:50:41.013368 16219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:41.014569 16204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:50:41.014709 16204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "555105c180d9494f925fca4824d05d0a"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "742f4fc6a640bb1b0040d2649acbb3b7"
server_key_iv: "497d730120cc5998859af570ff80524d"
server_key_version: "encryptionkey@0"
I20260927 22:50:41.014873 16204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:50:41.027905 16204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:50:41.031736 16204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:50:41.032006 16204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:50:41.032924 16204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:50:41.034402 16204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:50:41.034483 16204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:41.034617 16204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:50:41.034679 16204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:50:41.064571 16204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42009
I20260927 22:50:41.064616 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42009 every 8 connection(s)
I20260927 22:50:41.065757 16204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:50:41.073316 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16204
I20260927 22:50:41.073508 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:50:41.074067 388 external_mini_cluster.cc:1428] Setting key 5e0565ec8c6a91312a6af84eb0e1999d
Sep 27 22:50:41 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:50:41.084699 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.068095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59823 (local address 127.0.97.62:34779)
0927 22:50:41.068257 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.068261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.069385 (+ 1124us) server_negotiation.cc:407] Connection header received
0927 22:50:41.071129 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.071136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.071217 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.071306 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:41.072876 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.073952 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.075781 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.076318 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.076412 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.079937 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:41.079942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:41.079946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:41.079978 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:41.082196 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.082815 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.082819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.082822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.082930 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.083272 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.083275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.083277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.083563 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:41.083676 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.084292 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.084449 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:50:41.085645 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.068561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:59823)
0927 22:50:41.069227 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.069267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.070864 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.071579 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.071588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.072102 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:50:41.072709 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.072729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.074113 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.074118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.075639 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.075652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.076539 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.076547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.076838 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.077479 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:50:41.077504 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:50:41.079739 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:50:41.082320 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:41.082326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:41.082340 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:41.082728 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:41.083019 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:50:41.083022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:50:41.083024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:50:41.083196 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:50:41.083658 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:50:41.083664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:50:41.083970 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.084204 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.084584 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":252,"threads_started":1}
I20260927 22:50:41.086942 16332 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34779
I20260927 22:50:41.087236 16332 heartbeater.cc:464] Registering TS with master...
I20260927 22:50:41.088042 16332 heartbeater.cc:511] Master 127.0.97.62:34779 requested a full tablet report, sending...
I20260927 22:50:41.089421 15875 ts_manager.cc:194] Registered new tserver with Master: 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009)
I20260927 22:50:41.090166 15875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59823
I20260927 22:50:41.100929 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:50:41.117097 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.105061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.0.97.62:34779)
0927 22:50:41.105243 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.105250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.105703 (+ 453us) server_negotiation.cc:407] Connection header received
0927 22:50:41.105858 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.105861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.105917 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.105988 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:41.107360 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.108046 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.109409 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.109809 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.109891 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.112124 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:41.112128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:41.112131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:41.112157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:41.114407 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.115095 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.115100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.115102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.115232 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.115583 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.115587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.115590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.116044 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:41.116286 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.116558 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.116783 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:50:41.121222 15875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:50:41.127805 15875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:50:41.146770 16065 heartbeater.cc:503] Master 127.0.97.62:34779 was elected leader, sending a full tablet report...
I20260927 22:50:41.153514 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.139287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.0.97.3:42009)
0927 22:50:41.139944 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.139952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.144175 (+ 4223us) server_negotiation.cc:407] Connection header received
0927 22:50:41.144550 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.144557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.144793 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.144953 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:41.146390 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.148015 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.150969 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.152346 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.152473 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.152707 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.153004 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.153170 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":376,"threads_started":1}
I20260927 22:50:41.153584 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.138971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42009 (local address 127.0.0.1:57806)
0927 22:50:41.144014 (+ 5043us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.144075 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.144324 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.145313 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.145321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.145360 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:41.146240 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.146249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.148219 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.148225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.150824 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.150834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.152551 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.152557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.152748 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.152799 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.152938 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.153139 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4791,"mutex_wait_us":645,"spinlock_wait_cycles":14592,"thread_start_us":4652,"threads_started":1}
I20260927 22:50:41.153975 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.139731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.0.97.1:39491)
0927 22:50:41.140291 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.140296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.144176 (+ 3880us) server_negotiation.cc:407] Connection header received
0927 22:50:41.144550 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.144557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.144793 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.144997 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:41.146407 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.148065 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.151532 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.152445 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.152565 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.152795 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.153273 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.153432 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":310,"threads_started":1}
I20260927 22:50:41.154376 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.144965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40019 (local address 127.0.0.1:38662)
0927 22:50:41.145786 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.145802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.145902 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.146632 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.146636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.146659 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:41.147010 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.147022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.148720 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.148724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.150557 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.150570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.153456 (+ 2886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.153462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.153747 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.153783 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.153871 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.153992 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":21,"thread_start_us":522,"threads_started":1}
I20260927 22:50:41.154699 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.144627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.0.97.2:40019)
0927 22:50:41.146025 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.146031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.146052 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:50:41.146119 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.146124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.146279 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.146435 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:41.147183 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.148573 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.150699 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.153031 (+ 2332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.153142 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.153306 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.153919 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.154258 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":1140,"threads_started":1}
I20260927 22:50:41.154755 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.139167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49222)
0927 22:50:41.144014 (+ 4847us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.144075 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.144324 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.145313 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.145321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.145358 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:41.145805 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.145818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.148219 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.148225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.150820 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.150834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.152925 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.152931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.153086 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.153115 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.154221 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.154367 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4648,"mutex_wait_us":307,"spinlock_wait_cycles":6528,"thread_start_us":4445,"threads_started":1}
I20260927 22:50:41.157881 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 482b5d7746854784b52fbb3e92717c29 (DEFAULT_TABLE table=test-table [id=bb468106ef654280b35813ad3c5ce839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:41.158362 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 482b5d7746854784b52fbb3e92717c29 (DEFAULT_TABLE table=test-table [id=bb468106ef654280b35813ad3c5ce839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:41.159502 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482b5d7746854784b52fbb3e92717c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:41.159696 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482b5d7746854784b52fbb3e92717c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:41.161630 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 482b5d7746854784b52fbb3e92717c29 (DEFAULT_TABLE table=test-table [id=bb468106ef654280b35813ad3c5ce839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:50:41.162802 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482b5d7746854784b52fbb3e92717c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:50:41.173029 16350 tablet_bootstrap.cc:492] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Bootstrap starting.
I20260927 22:50:41.176102 16351 tablet_bootstrap.cc:492] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Bootstrap starting.
I20260927 22:50:41.177016 16350 tablet_bootstrap.cc:654] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:41.178243 16350 log.cc:824] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:41.179360 16351 tablet_bootstrap.cc:654] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:41.179657 16352 tablet_bootstrap.cc:492] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Bootstrap starting.
I20260927 22:50:41.180540 16351 log.cc:824] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:41.181236 16350 tablet_bootstrap.cc:492] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: No bootstrap required, opened a new log
I20260927 22:50:41.181514 16350 ts_tablet_manager.cc:1397] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:50:41.183043 16352 tablet_bootstrap.cc:654] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Neither blocks nor log segments found. Creating new log.
I20260927 22:50:41.183367 16351 tablet_bootstrap.cc:492] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: No bootstrap required, opened a new log
I20260927 22:50:41.183701 16351 ts_tablet_manager.cc:1397] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:50:41.184198 16352 log.cc:824] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:50:41.186947 16352 tablet_bootstrap.cc:492] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: No bootstrap required, opened a new log
I20260927 22:50:41.187192 16352 ts_tablet_manager.cc:1397] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 22:50:41.189284 16350 raft_consensus.cc:348] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.189601 16350 raft_consensus.cc:374] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:41.189690 16350 raft_consensus.cc:729] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db67e52227504cefa263cb9ed1987f97, State: Initialized, Role: FOLLOWER
I20260927 22:50:41.190199 16350 consensus_queue.cc:260] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.192296 16350 ts_tablet_manager.cc:1428] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 22:50:41.193130 16351 raft_consensus.cc:348] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.193542 16351 raft_consensus.cc:374] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:41.193643 16351 raft_consensus.cc:729] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4585d7a5c5048cea194b42a1de1c5ab, State: Initialized, Role: FOLLOWER
I20260927 22:50:41.194224 16351 consensus_queue.cc:260] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.196550 16199 heartbeater.cc:503] Master 127.0.97.62:34779 was elected leader, sending a full tablet report...
I20260927 22:50:41.197273 16352 raft_consensus.cc:348] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.197692 16352 raft_consensus.cc:374] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:50:41.197783 16352 raft_consensus.cc:729] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555105c180d9494f925fca4824d05d0a, State: Initialized, Role: FOLLOWER
I20260927 22:50:41.198433 16351 ts_tablet_manager.cc:1428] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Time spent starting tablet: real 0.015s user 0.000s sys 0.013s
I20260927 22:50:41.198350 16352 consensus_queue.cc:260] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.199363 16332 heartbeater.cc:503] Master 127.0.97.62:34779 was elected leader, sending a full tablet report...
I20260927 22:50:41.199873 16352 ts_tablet_manager.cc:1428] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
W20260927 22:50:41.317221 16333 tablet.cc:2369] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:50:41.344856 16200 tablet.cc:2369] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:50:41.369747 16066 tablet.cc:2369] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:50:41.384671 16356 raft_consensus.cc:482] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:50:41.384959 16356 raft_consensus.cc:504] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.386613 16356 leader_election.cc:288] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
I20260927 22:50:41.392617 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.387054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47791 (local address 127.0.97.3:42009)
0927 22:50:41.387231 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.387237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.387624 (+ 387us) server_negotiation.cc:407] Connection header received
0927 22:50:41.387800 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.387806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.387881 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.387977 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:41.388571 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.389304 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.390972 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.391787 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.391898 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.392000 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.392258 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.392378 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:50:41.392743 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.386866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42009 (local address 127.0.97.1:47791)
0927 22:50:41.387487 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.387521 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.387663 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.388097 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.388106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.388133 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:41.388442 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.388450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.389423 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.389429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.390831 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.390843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.391965 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.391970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.392117 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.392141 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.392220 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.392343 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":323,"threads_started":1}
I20260927 22:50:41.393297 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.387796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35609 (local address 127.0.97.2:40019)
0927 22:50:41.387977 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.387982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.388336 (+ 354us) server_negotiation.cc:407] Connection header received
0927 22:50:41.388492 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.388497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.388560 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.388629 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:41.389136 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.389804 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.391391 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.392258 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.392373 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.392465 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.392897 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.393032 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:50:41.393565 16357 raft_consensus.cc:482] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:50:41.393983 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.387590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40019 (local address 127.0.97.1:35609)
0927 22:50:41.388236 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.388251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.388363 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.388660 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.388665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.388686 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:41.388999 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.389008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.389935 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.389938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.391248 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.391260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.392465 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.392475 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.392675 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.392708 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.393549 (+ 841us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.393681 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":359,"threads_started":1}
I20260927 22:50:41.393834 16357 raft_consensus.cc:504] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.394026 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482b5d7746854784b52fbb3e92717c29" candidate_uuid: "db67e52227504cefa263cb9ed1987f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555105c180d9494f925fca4824d05d0a" is_pre_election: true
I20260927 22:50:41.394550 16287 raft_consensus.cc:2454] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db67e52227504cefa263cb9ed1987f97 in term 0.
I20260927 22:50:41.395046 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482b5d7746854784b52fbb3e92717c29" candidate_uuid: "db67e52227504cefa263cb9ed1987f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" is_pre_election: true
I20260927 22:50:41.395259 15956 leader_election.cc:302] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555105c180d9494f925fca4824d05d0a, db67e52227504cefa263cb9ed1987f97; no voters:
I20260927 22:50:41.395556 16154 raft_consensus.cc:2454] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db67e52227504cefa263cb9ed1987f97 in term 0.
I20260927 22:50:41.395709 16356 raft_consensus.cc:2781] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:50:41.395846 16357 leader_election.cc:288] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), db67e52227504cefa263cb9ed1987f97 (127.0.97.1:39491)
I20260927 22:50:41.395864 16356 raft_consensus.cc:482] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:50:41.395951 16356 raft_consensus.cc:3037] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:41.399387 16356 raft_consensus.cc:504] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.400120 16356 leader_election.cc:288] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 1 election: Requested vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
I20260927 22:50:41.400859 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482b5d7746854784b52fbb3e92717c29" candidate_uuid: "db67e52227504cefa263cb9ed1987f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555105c180d9494f925fca4824d05d0a"
I20260927 22:50:41.401078 16287 raft_consensus.cc:3037] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:41.401135 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482b5d7746854784b52fbb3e92717c29" candidate_uuid: "db67e52227504cefa263cb9ed1987f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4585d7a5c5048cea194b42a1de1c5ab"
I20260927 22:50:41.401333 16154 raft_consensus.cc:3037] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:50:41.404321 16287 raft_consensus.cc:2454] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db67e52227504cefa263cb9ed1987f97 in term 1.
I20260927 22:50:41.404799 16154 raft_consensus.cc:2454] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db67e52227504cefa263cb9ed1987f97 in term 1.
I20260927 22:50:41.404805 15956 leader_election.cc:302] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555105c180d9494f925fca4824d05d0a, db67e52227504cefa263cb9ed1987f97; no voters:
I20260927 22:50:41.405222 16356 raft_consensus.cc:2781] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:50:41.406803 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.396304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51321 (local address 127.0.97.1:39491)
0927 22:50:41.396457 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.396463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.400032 (+ 3569us) server_negotiation.cc:407] Connection header received
0927 22:50:41.400295 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.400300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.400369 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.400427 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:41.401225 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.402227 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.404914 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.405877 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.405960 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.406030 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.406447 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.406564 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 22:50:41.406885 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.396111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.97.2:51321)
0927 22:50:41.399946 (+ 3835us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.399968 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.400149 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.400612 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.400617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.400642 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:41.401059 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.401071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.402434 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.402439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.404745 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.404759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.406178 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.406183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.406300 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.406322 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.406393 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.406514 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3715,"spinlock_wait_cycles":24448,"thread_start_us":3558,"threads_started":1}
I20260927 22:50:41.407224 16356 raft_consensus.cc:686] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 1 LEADER]: Becoming Leader. State: Replica: db67e52227504cefa263cb9ed1987f97, State: Running, Role: LEADER
I20260927 22:50:41.407943 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.396557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46747 (local address 127.0.97.3:42009)
0927 22:50:41.396726 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.396732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.400703 (+ 3971us) server_negotiation.cc:407] Connection header received
0927 22:50:41.400968 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.400974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.401075 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.401137 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:50:41.401958 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.403123 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.405555 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.406832 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.406937 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.407049 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.407585 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.407711 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:50:41.407755 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482b5d7746854784b52fbb3e92717c29" candidate_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db67e52227504cefa263cb9ed1987f97" is_pre_election: true
I20260927 22:50:41.408236 16356 consensus_queue.cc:237] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:50:41.408588 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.396215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42009 (local address 127.0.97.2:46747)
0927 22:50:41.400615 (+ 4400us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.400632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:50:41.400814 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:50:41.401326 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:50:41.401330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:50:41.401354 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:50:41.401786 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.401799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.403332 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.403337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.405389 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:50:41.405402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.407223 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.407228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:50:41.407384 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.407414 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:50:41.408156 (+ 742us) client_negotiation.cc:240] Negotiation successful
0927 22:50:41.408286 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4286,"thread_start_us":4157,"threads_started":1}
I20260927 22:50:41.409128 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482b5d7746854784b52fbb3e92717c29" candidate_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555105c180d9494f925fca4824d05d0a" is_pre_election: true
I20260927 22:50:41.409389 16287 raft_consensus.cc:2379] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4585d7a5c5048cea194b42a1de1c5ab in current term 1: Already voted for candidate db67e52227504cefa263cb9ed1987f97 in this term.
I20260927 22:50:41.409943 16090 leader_election.cc:302] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4585d7a5c5048cea194b42a1de1c5ab; no voters: 555105c180d9494f925fca4824d05d0a, db67e52227504cefa263cb9ed1987f97
I20260927 22:50:41.410323 16357 raft_consensus.cc:2726] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:50:41.413898 15875 catalog_manager.cc:5495] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 reported cstate change: term changed from 0 to 1, leader changed from <none> to db67e52227504cefa263cb9ed1987f97 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "db67e52227504cefa263cb9ed1987f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } health_report { overall_health: UNKNOWN } } }
I20260927 22:50:41.449297 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.444040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.0.97.1:39491)
0927 22:50:41.444271 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.444276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.444295 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:50:41.444492 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.444496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.444591 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.444722 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:50:41.445092 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.445803 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.447229 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.447657 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.447762 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.448257 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:50:41.448729 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:50:41.448870 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.448940 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.449020 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 22:50:41.457005 16287 raft_consensus.cc:1261] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Refusing update from remote peer db67e52227504cefa263cb9ed1987f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:41.457051 16154 raft_consensus.cc:1261] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Refusing update from remote peer db67e52227504cefa263cb9ed1987f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:50:41.457846 16356 consensus_queue.cc:1035] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:41.458153 16363 consensus_queue.cc:1035] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:50:41.467839 16366 mvcc.cc:205] Tried to move back new op lower bound from 7334090512198352896 to 7334090512013209600. Current Snapshot: MvccSnapshot[applied={T|T < 7334090512198352896}]
I20260927 22:50:41.483310 15874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:41.483659 15874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:50:41.487200 15874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Sending DeleteTablet for 3 replicas of tablet 482b5d7746854784b52fbb3e92717c29
I20260927 22:50:41.488389 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet 482b5d7746854784b52fbb3e92717c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49222
I20260927 22:50:41.488415 16267 tablet_service.cc:1514] Processing DeleteTablet for tablet 482b5d7746854784b52fbb3e92717c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57806
I20260927 22:50:41.488415 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 482b5d7746854784b52fbb3e92717c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:50:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38662
I20260927 22:50:41.489516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 15937
I20260927 22:50:41.514644 16376 tablet_replica.cc:330] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: stopping tablet replica
I20260927 22:50:41.515206 16376 raft_consensus.cc:2229] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:50:41.515890 16377 tablet_replica.cc:330] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: stopping tablet replica
I20260927 22:50:41.516353 16377 raft_consensus.cc:2229] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:41.516881 16377 raft_consensus.cc:2258] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:41.517777 16376 raft_consensus.cc:2258] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:41.520084 16377 ts_tablet_manager.cc:1905] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:41.525429 16376 ts_tablet_manager.cc:1905] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:50:41 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:50:41.530761 16378 tablet_replica.cc:330] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: stopping tablet replica
I20260927 22:50:41.531195 16378 raft_consensus.cc:2229] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:50:41.531697 16378 raft_consensus.cc:2258] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:50:41.533635 16378 ts_tablet_manager.cc:1905] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:50:41.534045 16377 ts_tablet_manager.cc:1918] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:41.534211 16377 log.cc:1196] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/482b5d7746854784b52fbb3e92717c29
I20260927 22:50:41.534693 16377 ts_tablet_manager.cc:1939] T 482b5d7746854784b52fbb3e92717c29 P 555105c180d9494f925fca4824d05d0a: Deleting consensus metadata
I20260927 22:50:41.536550 15862 catalog_manager.cc:4841] TS 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009): tablet 482b5d7746854784b52fbb3e92717c29 (table test-table [id=bb468106ef654280b35813ad3c5ce839]) successfully deleted
I20260927 22:50:41.537393 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.490579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.0.97.1:39491)
0927 22:50:41.490759 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.490765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.513365 (+ 22600us) server_negotiation.cc:407] Connection header received
0927 22:50:41.513584 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.513591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.513729 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.513854 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:41.516601 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.517908 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.520651 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.521381 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.521498 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.527802 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:41.527820 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:41.527840 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:41.527904 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:41.532925 (+ 5021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.533935 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.533943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.533946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.534191 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.534817 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.534823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.534826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.535296 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:41.535599 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.536980 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.537160 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:50:41.538085 16060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:41.540212 16378 ts_tablet_manager.cc:1918] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:50:41.540369 16378 log.cc:1196] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/482b5d7746854784b52fbb3e92717c29
I20260927 22:50:41.540843 16378 ts_tablet_manager.cc:1939] T 482b5d7746854784b52fbb3e92717c29 P f4585d7a5c5048cea194b42a1de1c5ab: Deleting consensus metadata
I20260927 22:50:41.542498 15862 catalog_manager.cc:4841] TS f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019): tablet 482b5d7746854784b52fbb3e92717c29 (table test-table [id=bb468106ef654280b35813ad3c5ce839]) successfully deleted
I20260927 22:50:41.640097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 15937
I20260927 22:50:41.655215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 16071
W20260927 22:50:41.655222 15859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39491 (error 108)
I20260927 22:50:41.657687 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.657056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49254)
0927 22:50:41.657431 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.657524 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":189,"threads_started":1}
W20260927 22:50:41.657881 15859 catalog_manager.cc:4584] TS db67e52227504cefa263cb9ed1987f97 (127.0.97.1:39491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 482b5d7746854784b52fbb3e92717c29: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Sep 27 22:50:41 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:50:41.670055 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.655980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.97.2:40019)
0927 22:50:41.656317 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.656323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.656345 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:50:41.656412 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.656416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.656504 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.656631 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:41.658354 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.659076 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.660505 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.660900 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.660988 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.665114 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:41.665126 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:41.665140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:41.665174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:41.667786 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.668430 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.668436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.668438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.668551 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.668848 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.668851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.668853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.669186 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:41.669386 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.669663 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.669815 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260927 22:50:41.670631 16194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:41.711175 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.710725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49256)
0927 22:50:41.710920 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.711025 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:50:41.712124 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.711779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49268)
0927 22:50:41.711919 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.711986 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:50:41.760643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 16071
I20260927 22:50:41.772446 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.771941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49274)
0927 22:50:41.772170 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.772277 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:50:41.773485 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.773092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49276)
0927 22:50:41.773241 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.773323 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:50:41.774212 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 16204
Sep 27 22:50:41 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:50:41.788308 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.774848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.0.97.3:42009)
0927 22:50:41.775023 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:50:41.775030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:50:41.775097 (+ 67us) server_negotiation.cc:407] Connection header received
0927 22:50:41.775318 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:50:41.775324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:50:41.775398 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:50:41.775514 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:50:41.776807 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.777477 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.778677 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:50:41.779079 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:50:41.779184 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:50:41.783238 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:50:41.783252 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:50:41.783269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:50:41.783300 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:50:41.785778 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.786403 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.786411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.786413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.786566 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:50:41.786940 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:50:41.786944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:50:41.786946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:50:41.787313 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:50:41.787579 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:50:41.787784 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:50:41.787971 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:50:41.788879 16327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:41.854223 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.853772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49280)
0927 22:50:41.853973 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.854076 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:50:41.855345 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:50:41.854960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:49286)
0927 22:50:41.855143 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:50:41.855221 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:50:41.876544 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 16204
I20260927 22:50:41.891466 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 15843
I20260927 22:50:41.892588 15905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:50:41.999176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 15843
I20260927 22:51:02.016799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34779
--webserver_interface=127.0.97.62
--webserver_port=42911
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:02.340126 16386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:02.340435 16386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:02.340567 16386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:02.347847 16386 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:51:02.347950 16386 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:51:02.348016 16386 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:51:02.348085 16386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:02.348161 16386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:02.348256 16386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:02.348313 16386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:02.348393 16386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:02.358259 16386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:02.359558 16386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:02.360852 16386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:02.382156 16395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:02.382156 16393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:02.382169 16392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:02.382941 16386 server_base.cc:1034] running on GCE node
I20260927 22:51:02.383822 16386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:02.385305 16386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:02.386541 16386 hybrid_clock.cc:648] HybridClock initialized: now 1790549462386452 us; error 41 us; skew 500 ppm
I20260927 22:51:02.386911 16386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:02 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:02.394435 16386 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:02.396075 16386 webserver.cc:460] Webserver started at http://127.0.97.62:42911/ using document root <none> and password file <none>
I20260927 22:51:02.396834 16386 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:02.396936 16386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:02.402714 16386 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:51:02.405625 16402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:02.407043 16386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:02.407196 16386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "24f8d4caeee54e89940b5fffc77edf0c"
format_stamp: "Formatted at 2026-09-27 22:50:39 on dist-test-slave-kr25"
server_key: "32bead642361ba73cb8a8534c1e95d24"
server_key_iv: "55fca012781e3f4b0c9e7584d246405b"
server_key_version: "encryptionkey@0"
I20260927 22:51:02.408282 16386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:02.431604 16386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:02.436090 16386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:02.436363 16386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:02.453877 16386 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34779
I20260927 22:51:02.453897 16453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34779 every 8 connection(s)
I20260927 22:51:02.455468 16386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:02.458278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16386
I20260927 22:51:02.459029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39491
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34779
--webserver_port=41801
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:51:02.462530 16454 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:51:02.465755 16454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Bootstrap starting.
I20260927 22:51:02.485013 16454 log.cc:824] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:02.496994 16454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:51:02.497541 16454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Bootstrap complete.
I20260927 22:51:02.506739 16454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:02.507637 16454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f8d4caeee54e89940b5fffc77edf0c, State: Initialized, Role: FOLLOWER
I20260927 22:51:02.508213 16454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:02.508435 16454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:02.508543 16454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:02.508699 16454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:51:02.512550 16454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:02.513088 16454 leader_election.cc:302] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f8d4caeee54e89940b5fffc77edf0c; no voters:
I20260927 22:51:02.513708 16454 leader_election.cc:288] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:51:02.513840 16458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:51:02.514921 16458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 LEADER]: Becoming Leader. State: Replica: 24f8d4caeee54e89940b5fffc77edf0c, State: Running, Role: LEADER
I20260927 22:51:02.515494 16458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:02.515904 16454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:02.517817 16459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "24f8d4caeee54e89940b5fffc77edf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } } }
I20260927 22:51:02.518181 16459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:02.521920 16460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f8d4caeee54e89940b5fffc77edf0c. Latest consensus state: current_term: 2 leader_uuid: "24f8d4caeee54e89940b5fffc77edf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } } }
I20260927 22:51:02.522238 16460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:02.523787 16463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:02.532122 16463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:02.533609 16463 catalog_manager.cc:1259] Loaded cluster ID: 0bea33be67514e8798d611dea82b50df
I20260927 22:51:02.533778 16463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:02.538089 16463 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:02.539825 16463 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Loaded TSK: 0
I20260927 22:51:02.561630 16475 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:51:02.580626 16463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Generated new TSK 1
I20260927 22:51:02.581135 16463 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:51:02.800498 16456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:02.800766 16456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:02.800899 16456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:02.808138 16456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:02.808244 16456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:02.808382 16456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:02.818550 16456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41801
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:02.819906 16456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:02.821211 16456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:02.833983 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:02.833963 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:02.835994 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:02.841025 16456 server_base.cc:1034] running on GCE node
I20260927 22:51:02.841640 16456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:02.842609 16456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:02.843807 16456 hybrid_clock.cc:648] HybridClock initialized: now 1790549462843740 us; error 64 us; skew 500 ppm
I20260927 22:51:02.844121 16456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:02 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:02.847095 16456 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:02.848695 16456 webserver.cc:460] Webserver started at http://127.0.97.1:41801/ using document root <none> and password file <none>
I20260927 22:51:02.849400 16456 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:02.849517 16456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:02.855173 16456 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:51:02.858273 16491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:02.859740 16456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:51:02.859887 16456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "db67e52227504cefa263cb9ed1987f97"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "be7a5aa00173983e0a25622f27ef8324"
server_key_iv: "340a54aae8ac335579f81d995f774054"
server_key_version: "encryptionkey@0"
I20260927 22:51:02.860868 16456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:02.876305 16456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:02.880470 16456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:02.880707 16456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:02.881721 16456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:02.883605 16498 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 22:51:02.888675 16498 ts_tablet_manager.cc:1852] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 22:51:02.889607 16498 ts_tablet_manager.cc:1905] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:02.896790 16498 ts_tablet_manager.cc:1918] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:02.896975 16498 log.cc:1196] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/482b5d7746854784b52fbb3e92717c29
I20260927 22:51:02.897420 16498 ts_tablet_manager.cc:1939] T 482b5d7746854784b52fbb3e92717c29 P db67e52227504cefa263cb9ed1987f97: Deleting consensus metadata
I20260927 22:51:02.897783 16456 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 22:51:02.897884 16456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260927 22:51:02.898006 16456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:02.898069 16456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:02.928751 16456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39491
I20260927 22:51:02.928800 16604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39491 every 8 connection(s)
I20260927 22:51:02.930039 16456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:02.932274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16456
I20260927 22:51:02.933009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40019
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34779
--webserver_port=34275
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:51:02 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:02.962293 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:02.933360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:33767)
0927 22:51:02.934338 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:02.934410 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:02.937195 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:02.941522 (+ 4327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:02.941535 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:02.942274 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:02.943026 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:02.943050 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:02.944643 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:02.944649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:02.946964 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:02.946980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:02.948799 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:02.948807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:02.949071 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:02.949751 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:02.949780 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:02.952867 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:02.957053 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:02.957063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:02.957079 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:02.957658 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:02.958162 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:02.958168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:02.958193 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:02.958412 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:02.959526 (+ 1114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:02.959534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:02.959978 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 22:51:02.960303 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 22:51:02.960864 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":362,"threads_started":1}
I20260927 22:51:02.964921 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:02.932944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33767 (local address 127.0.97.62:34779)
0927 22:51:02.940217 (+ 7273us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:02.940229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:02.940272 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:51:02.940592 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:02.940607 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:02.940978 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:02.941288 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:02.943216 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:02.944487 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:02.947522 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:02.948567 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:02.948760 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:02.953214 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:02.953225 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:02.953246 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:02.953297 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:02.956871 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:02.957796 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:02.957822 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:02.957832 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:02.958038 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:02.958533 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:02.958538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:02.958542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:02.959213 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:02.959408 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:02.962959 (+ 3551us) server_negotiation.cc:299] Negotiation successful
0927 22:51:02.963382 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7037,"thread_start_us":6930,"threads_started":1}
I20260927 22:51:02.968173 16605 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34779
I20260927 22:51:02.968640 16605 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:02.969884 16605 heartbeater.cc:511] Master 127.0.97.62:34779 requested a full tablet report, sending...
I20260927 22:51:02.972494 16419 ts_manager.cc:194] Registered new tserver with Master: db67e52227504cefa263cb9ed1987f97 (127.0.97.1:39491)
I20260927 22:51:02.974673 16419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33767
W20260927 22:51:03.269091 16609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:03.269371 16609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:03.269472 16609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:03.276618 16609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:03.276722 16609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:03.276866 16609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:03.286329 16609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34275
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:03.287542 16609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:03.288789 16609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:03.301280 16614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:03.301247 16615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:03.301342 16617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:03.302000 16609 server_base.cc:1034] running on GCE node
I20260927 22:51:03.302572 16609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:03.303503 16609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:03.304692 16609 hybrid_clock.cc:648] HybridClock initialized: now 1790549463304628 us; error 59 us; skew 500 ppm
I20260927 22:51:03.304983 16609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:03 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:03.307895 16609 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:03.309226 16609 webserver.cc:460] Webserver started at http://127.0.97.2:34275/ using document root <none> and password file <none>
I20260927 22:51:03.309834 16609 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:03.309937 16609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:03.314905 16609 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:51:03.317432 16624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:03.318696 16609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:03.318826 16609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "f4585d7a5c5048cea194b42a1de1c5ab"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "b3166e8f2bb99ffe773d322a921564a0"
server_key_iv: "080d9168a44f02cbcf8adf1aecce729b"
server_key_version: "encryptionkey@0"
I20260927 22:51:03.319761 16609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:03.334154 16609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:03.338351 16609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:03.338581 16609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:03.339497 16609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:03.340719 16609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:03.340816 16609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:03.340896 16609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:03.340945 16609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:03.370223 16609 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40019
I20260927 22:51:03.370280 16736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40019 every 8 connection(s)
I20260927 22:51:03.371685 16609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:03.378630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16609
I20260927 22:51:03.379567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42009
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34779
--webserver_port=43265
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:51:03 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:03.391747 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.374212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42029 (local address 127.0.97.62:34779)
0927 22:51:03.374403 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.374408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.375592 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 22:51:03.377373 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.377378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.377448 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.377540 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:03.379347 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.380146 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.382217 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.382650 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.382742 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.386372 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:03.386377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:03.386381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:03.386414 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:03.388904 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.389573 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.389577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.389580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.389702 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.390103 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.390108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.390111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.390377 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:03.390517 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.391161 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.391390 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:51:03.392680 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.374701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:42029)
0927 22:51:03.375397 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:03.375477 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:03.377126 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:03.377738 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:03.377747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:03.378346 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:03.379163 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.379189 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.380305 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.380311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.382098 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.382111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.382846 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.382852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.383108 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.383792 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:03.383818 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:03.386155 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:03.389042 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:03.389051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:03.389066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:03.389458 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:03.389802 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:03.389806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:03.389810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:03.390000 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:03.390495 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:03.390502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:03.390819 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 22:51:03.391059 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 22:51:03.391603 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":287,"threads_started":1}
I20260927 22:51:03.394065 16737 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34779
I20260927 22:51:03.394380 16737 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:03.395221 16737 heartbeater.cc:511] Master 127.0.97.62:34779 requested a full tablet report, sending...
I20260927 22:51:03.396801 16419 ts_manager.cc:194] Registered new tserver with Master: f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
I20260927 22:51:03.397902 16419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42029
W20260927 22:51:03.700659 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:03.700928 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:03.701021 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:03.708127 16740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:03.708223 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:03.708348 16740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:03.717917 16740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43265
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:03.719054 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:03.720281 16740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:03.732442 16745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:03.732412 16746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:03.732486 16748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:03.733291 16740 server_base.cc:1034] running on GCE node
I20260927 22:51:03.733855 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:03.734722 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:03.735901 16740 hybrid_clock.cc:648] HybridClock initialized: now 1790549463735835 us; error 60 us; skew 500 ppm
I20260927 22:51:03.736189 16740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:03 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:03.739135 16740 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:03.740566 16740 webserver.cc:460] Webserver started at http://127.0.97.3:43265/ using document root <none> and password file <none>
I20260927 22:51:03.741204 16740 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:03.741309 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:03.746357 16740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:51:03.748829 16755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:03.750105 16740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:03.750229 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "555105c180d9494f925fca4824d05d0a"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "742f4fc6a640bb1b0040d2649acbb3b7"
server_key_iv: "497d730120cc5998859af570ff80524d"
server_key_version: "encryptionkey@0"
I20260927 22:51:03.751113 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:03.769259 16740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:03.773336 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:03.773564 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:03.774513 16740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:03.775874 16740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:03.775969 16740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:03.776053 16740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:03.776113 16740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:03.805881 16740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42009
I20260927 22:51:03.805932 16867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42009 every 8 connection(s)
I20260927 22:51:03.807296 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:03.811084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16740
Sep 27 22:51:03 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:03.828014 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.812030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.97.62:34779)
0927 22:51:03.812513 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.812518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.812889 (+ 371us) server_negotiation.cc:407] Connection header received
0927 22:51:03.813069 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.813073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.813137 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.813225 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:03.815180 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.816229 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.818208 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.818630 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.818755 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.822035 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:03.822042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:03.822046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:03.822083 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:03.825553 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.826163 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.826168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.826172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.826307 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.826677 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.826681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.826683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.826927 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:03.827168 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.827382 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.827593 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
I20260927 22:51:03.829372 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.809732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36987 (local address 127.0.97.62:34779)
0927 22:51:03.809914 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.809919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.811106 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 22:51:03.813533 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.813536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.813588 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.813663 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:03.815636 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.816636 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.818671 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.819154 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.819258 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.822928 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:03.822934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:03.822938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:03.822985 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:03.826330 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.827060 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.827064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.827067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.827187 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.827687 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.827692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.827695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.828027 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:03.828159 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.828905 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.829093 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:51:03.830477 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.810183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:36987)
0927 22:51:03.810932 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:03.810982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:03.813223 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:03.813980 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:03.813993 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:03.814714 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:03.815486 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.815508 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.816780 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.816786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.818553 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.818565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.819367 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.819374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.819710 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.820242 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:03.820277 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:03.822748 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:03.826473 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:03.826482 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:03.826498 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:03.826946 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:03.827292 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:03.827297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:03.827301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:03.827563 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:03.828161 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:03.828169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:03.828519 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 22:51:03.828775 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 22:51:03.829168 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260927 22:51:03.831715 16868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34779
I20260927 22:51:03.831972 16868 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:03.832597 16868 heartbeater.cc:511] Master 127.0.97.62:34779 requested a full tablet report, sending...
I20260927 22:51:03.833797 16419 ts_manager.cc:194] Registered new tserver with Master: 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009)
I20260927 22:51:03.834550 16419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36987
I20260927 22:51:03.842625 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:03.848790 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.843484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.0.97.62:34779)
0927 22:51:03.843626 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.843632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.844040 (+ 408us) server_negotiation.cc:407] Connection header received
0927 22:51:03.844253 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.844257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.844383 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.844497 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:03.844918 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.845626 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.846979 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.847394 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.847543 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.848034 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:03.848226 (+ 192us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 22:51:03.848561 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42776: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 22:51:03.848886 16607 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42776: token expired
W20260927 22:51:03.849386 16338 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:34779)
I20260927 22:51:03.860813 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.849787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.97.62:34779)
0927 22:51:03.849910 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.849914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.849931 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:51:03.850051 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.850054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.850109 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.850178 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:03.851369 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.852111 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.853404 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.853813 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.853914 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.856459 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:03.856464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:03.856467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:03.856498 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:03.858739 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.859311 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.859315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.859317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.859451 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:03.859720 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:03.859723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:03.859725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:03.860027 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:03.860201 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.860399 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.860571 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 22:51:03.863958 16419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:03.871048 16419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:03.892001 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.882015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46774)
0927 22:51:03.882765 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:03.882806 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:03.882970 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:03.884070 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:03.884077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:03.884108 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:03.884561 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.884575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.886553 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.886558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.889277 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.889286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.890881 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.890887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.891108 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.891171 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 22:51:03.891309 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:51:03.891528 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":98,"spinlock_wait_cycles":13056,"thread_start_us":412,"threads_started":1}
I20260927 22:51:03.892379 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.882744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.0.97.3:42009)
0927 22:51:03.883582 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.883589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.884107 (+ 518us) server_negotiation.cc:407] Connection header received
0927 22:51:03.884386 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.884394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.884645 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.884797 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:03.885505 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.886822 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.889383 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.890434 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.890556 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.890740 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.891847 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.892037 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":491,"threads_started":1}
I20260927 22:51:03.892459 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.884076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40019 (local address 127.0.0.1:36106)
0927 22:51:03.884700 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:03.884718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:03.884835 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:03.885948 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:03.885954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:03.885979 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:03.886287 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.886297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.887928 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.887934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.889610 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.889624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.891639 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.891645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.891821 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.891853 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:51:03.891942 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:51:03.892071 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":72,"thread_start_us":455,"threads_started":1}
I20260927 22:51:03.892596 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.882058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42009 (local address 127.0.0.1:52142)
0927 22:51:03.884018 (+ 1960us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:03.884034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:03.884201 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:03.885007 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:03.885011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:03.885033 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:03.885351 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.885361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.886985 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.886989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.889220 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.889240 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.891270 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.891276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.891462 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.891494 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:51:03.891770 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 22:51:03.891916 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1842,"mutex_wait_us":71,"thread_start_us":1701,"threads_started":1}
I20260927 22:51:03.893255 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.882059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.0.97.1:39491)
0927 22:51:03.883396 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.883405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.883454 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:51:03.883555 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.883563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.883821 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.884852 (+ 1031us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:03.885053 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.886441 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.889423 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.890676 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.890804 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.891005 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.892758 (+ 1753us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.892962 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":1050,"threads_started":1}
I20260927 22:51:03.894135 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.884397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.97.2:40019)
0927 22:51:03.885411 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.885419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.885458 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:51:03.885547 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.885554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.885759 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.885930 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:03.886449 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.887779 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.889769 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.891126 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.891244 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.891491 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.893664 (+ 2173us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.893848 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":711,"threads_started":1}
I20260927 22:51:03.895058 16540 tablet_service.cc:1467] Processing CreateTablet for tablet 56b1ce0fba944905bfac0227523683cd (DEFAULT_TABLE table=test-table [id=bea53626f30240abb7ff290c39238452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:03.896615 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b1ce0fba944905bfac0227523683cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:03.896438 16803 tablet_service.cc:1467] Processing CreateTablet for tablet 56b1ce0fba944905bfac0227523683cd (DEFAULT_TABLE table=test-table [id=bea53626f30240abb7ff290c39238452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:03.897624 16803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b1ce0fba944905bfac0227523683cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:03.897684 16672 tablet_service.cc:1467] Processing CreateTablet for tablet 56b1ce0fba944905bfac0227523683cd (DEFAULT_TABLE table=test-table [id=bea53626f30240abb7ff290c39238452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:03.898875 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b1ce0fba944905bfac0227523683cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:03.906320 16884 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Bootstrap starting.
I20260927 22:51:03.910213 16885 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Bootstrap starting.
I20260927 22:51:03.910786 16884 tablet_bootstrap.cc:654] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:03.911844 16884 log.cc:824] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:03.913090 16887 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Bootstrap starting.
I20260927 22:51:03.913869 16885 tablet_bootstrap.cc:654] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:03.914892 16885 log.cc:824] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:03.915047 16884 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: No bootstrap required, opened a new log
I20260927 22:51:03.915292 16884 ts_tablet_manager.cc:1397] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:51:03.917142 16887 tablet_bootstrap.cc:654] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:03.917407 16885 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: No bootstrap required, opened a new log
I20260927 22:51:03.917635 16885 ts_tablet_manager.cc:1397] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260927 22:51:03.918251 16887 log.cc:824] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:03.920928 16887 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: No bootstrap required, opened a new log
I20260927 22:51:03.921542 16887 ts_tablet_manager.cc:1397] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:51:03.924985 16884 raft_consensus.cc:348] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.925107 16885 raft_consensus.cc:348] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.925386 16884 raft_consensus.cc:374] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:03.925411 16885 raft_consensus.cc:374] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:03.925490 16885 raft_consensus.cc:729] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555105c180d9494f925fca4824d05d0a, State: Initialized, Role: FOLLOWER
I20260927 22:51:03.925487 16884 raft_consensus.cc:729] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db67e52227504cefa263cb9ed1987f97, State: Initialized, Role: FOLLOWER
I20260927 22:51:03.926003 16885 consensus_queue.cc:260] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.926033 16884 consensus_queue.cc:260] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.927654 16885 ts_tablet_manager.cc:1428] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260927 22:51:03.927747 16868 heartbeater.cc:503] Master 127.0.97.62:34779 was elected leader, sending a full tablet report...
I20260927 22:51:03.928473 16605 heartbeater.cc:503] Master 127.0.97.62:34779 was elected leader, sending a full tablet report...
I20260927 22:51:03.930670 16884 ts_tablet_manager.cc:1428] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260927 22:51:03.931974 16606 tablet.cc:2369] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:03.935326 16887 raft_consensus.cc:348] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.935820 16887 raft_consensus.cc:374] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:03.935912 16887 raft_consensus.cc:729] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4585d7a5c5048cea194b42a1de1c5ab, State: Initialized, Role: FOLLOWER
I20260927 22:51:03.936538 16887 consensus_queue.cc:260] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.937588 16737 heartbeater.cc:503] Master 127.0.97.62:34779 was elected leader, sending a full tablet report...
I20260927 22:51:03.938279 16887 ts_tablet_manager.cc:1428] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Time spent starting tablet: real 0.017s user 0.009s sys 0.003s
I20260927 22:51:03.944306 16892 raft_consensus.cc:482] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:03.944533 16892 raft_consensus.cc:504] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.945940 16892 leader_election.cc:288] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db67e52227504cefa263cb9ed1987f97 (127.0.97.1:39491), 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009)
I20260927 22:51:03.952072 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.946414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59485 (local address 127.0.97.3:42009)
0927 22:51:03.946627 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.946634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.947059 (+ 425us) server_negotiation.cc:407] Connection header received
0927 22:51:03.947239 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.947243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.947300 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.947371 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:03.947960 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.948674 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.950425 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.951171 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.951280 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.951376 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.951722 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.951840 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:51:03.952149 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.946396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38517 (local address 127.0.97.1:39491)
0927 22:51:03.946581 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.946586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.947420 (+ 834us) server_negotiation.cc:407] Connection header received
0927 22:51:03.947586 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.947590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.947644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.947713 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:03.948221 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.948930 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.950580 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.951287 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.951392 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.951524 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.951782 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.951906 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:03.952258 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.946238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.97.2:38517)
0927 22:51:03.947316 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:03.947330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:03.947467 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:03.947741 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:03.947749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:03.947775 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:03.948077 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.948085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.949048 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.949053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.950441 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.950452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.951501 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.951507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.951648 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.951671 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:51:03.951747 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 22:51:03.951908 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":39,"thread_start_us":475,"threads_started":1}
I20260927 22:51:03.952546 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.946165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42009 (local address 127.0.97.2:59485)
0927 22:51:03.946957 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:03.946975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:03.947103 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:03.947400 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:03.947404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:03.947510 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:03.947824 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.947839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.948800 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.948805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.950288 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:03.950300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.951363 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.951370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:03.951579 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.951607 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:51:03.951686 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:51:03.951831 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":4,"spinlock_wait_cycles":22272,"thread_start_us":898,"threads_started":1}
I20260927 22:51:03.953122 16560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b1ce0fba944905bfac0227523683cd" candidate_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db67e52227504cefa263cb9ed1987f97" is_pre_election: true
I20260927 22:51:03.953467 16823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b1ce0fba944905bfac0227523683cd" candidate_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555105c180d9494f925fca4824d05d0a" is_pre_election: true
I20260927 22:51:03.953672 16560 raft_consensus.cc:2454] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4585d7a5c5048cea194b42a1de1c5ab in term 0.
I20260927 22:51:03.953950 16823 raft_consensus.cc:2454] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4585d7a5c5048cea194b42a1de1c5ab in term 0.
I20260927 22:51:03.954375 16625 leader_election.cc:302] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db67e52227504cefa263cb9ed1987f97, f4585d7a5c5048cea194b42a1de1c5ab; no voters:
I20260927 22:51:03.954736 16892 raft_consensus.cc:2781] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:03.954865 16892 raft_consensus.cc:482] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:03.954963 16892 raft_consensus.cc:3037] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:03.957086 16892 raft_consensus.cc:504] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.957631 16892 leader_election.cc:288] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [CANDIDATE]: Term 1 election: Requested vote from peers db67e52227504cefa263cb9ed1987f97 (127.0.97.1:39491), 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009)
I20260927 22:51:03.958017 16560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b1ce0fba944905bfac0227523683cd" candidate_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db67e52227504cefa263cb9ed1987f97"
I20260927 22:51:03.958055 16823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b1ce0fba944905bfac0227523683cd" candidate_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555105c180d9494f925fca4824d05d0a"
I20260927 22:51:03.958178 16560 raft_consensus.cc:3037] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:03.958216 16823 raft_consensus.cc:3037] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:03.960330 16560 raft_consensus.cc:2454] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4585d7a5c5048cea194b42a1de1c5ab in term 1.
I20260927 22:51:03.960350 16823 raft_consensus.cc:2454] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4585d7a5c5048cea194b42a1de1c5ab in term 1.
I20260927 22:51:03.960711 16625 leader_election.cc:302] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 555105c180d9494f925fca4824d05d0a, db67e52227504cefa263cb9ed1987f97, f4585d7a5c5048cea194b42a1de1c5ab; no voters:
I20260927 22:51:03.960980 16892 raft_consensus.cc:2781] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:03.961365 16892 raft_consensus.cc:686] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 1 LEADER]: Becoming Leader. State: Replica: f4585d7a5c5048cea194b42a1de1c5ab, State: Running, Role: LEADER
I20260927 22:51:03.961782 16892 consensus_queue.cc:237] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:03.966277 16419 catalog_manager.cc:5495] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } health_report { overall_health: HEALTHY } } }
I20260927 22:51:03.981696 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:03.976622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.0.97.2:40019)
0927 22:51:03.976783 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:03.976788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:03.976804 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:51:03.976987 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:03.976991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:03.977069 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:03.977192 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:03.977476 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.978145 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.979531 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:03.980002 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:03.980100 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:03.980624 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:03.981142 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:03.981251 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:03.981431 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 22:51:03.981501 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:51:03.989598 16560 raft_consensus.cc:1261] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Refusing update from remote peer f4585d7a5c5048cea194b42a1de1c5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:03.989598 16823 raft_consensus.cc:1261] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Refusing update from remote peer f4585d7a5c5048cea194b42a1de1c5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:03.990459 16895 consensus_queue.cc:1035] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:03.990760 16892 consensus_queue.cc:1035] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:04.001772 16898 mvcc.cc:205] Tried to move back new op lower bound from 7334090604491997184 to 7334090604391231488. Current Snapshot: MvccSnapshot[applied={T|T < 7334090604491997184}]
I20260927 22:51:04.015972 16418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:04.016270 16418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:04.019963 16418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Sending DeleteTablet for 3 replicas of tablet 56b1ce0fba944905bfac0227523683cd
I20260927 22:51:04.021147 16540 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b1ce0fba944905bfac0227523683cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46774
I20260927 22:51:04.021335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 16456
I20260927 22:51:04.021531 16672 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b1ce0fba944905bfac0227523683cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36106
I20260927 22:51:04.021899 16803 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b1ce0fba944905bfac0227523683cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52142
I20260927 22:51:04.042892 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.022961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.0.97.1:39491)
0927 22:51:04.023181 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:04.023188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:04.023206 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:51:04.023266 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:04.023271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:04.023370 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:04.023488 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:04.025724 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:04.026983 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:04.029091 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:04.029779 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:04.029898 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:04.034143 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:04.034166 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:04.034188 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:04.034235 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:04.038414 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:04.039453 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:04.039466 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:04.039470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:04.039636 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:04.040112 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:04.040116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:04.040120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:04.040618 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:04.040926 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:04.042218 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0927 22:51:04.042498 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 22:51:04.043684 16600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:04.056208 16910 tablet_replica.cc:330] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: stopping tablet replica
I20260927 22:51:04.057680 16910 raft_consensus.cc:2229] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:04.058276 16910 raft_consensus.cc:2258] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:04.059190 16909 tablet_replica.cc:330] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: stopping tablet replica
I20260927 22:51:04.059574 16909 raft_consensus.cc:2229] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:04.060115 16909 raft_consensus.cc:2258] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:04.062080 16910 ts_tablet_manager.cc:1905] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:04.064772 16909 ts_tablet_manager.cc:1905] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:04.068326 16910 ts_tablet_manager.cc:1918] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:04.068480 16910 log.cc:1196] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/56b1ce0fba944905bfac0227523683cd
I20260927 22:51:04.069016 16910 ts_tablet_manager.cc:1939] T 56b1ce0fba944905bfac0227523683cd P 555105c180d9494f925fca4824d05d0a: Deleting consensus metadata
I20260927 22:51:04.070832 16406 catalog_manager.cc:4841] TS 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009): tablet 56b1ce0fba944905bfac0227523683cd (table test-table [id=bea53626f30240abb7ff290c39238452]) successfully deleted
I20260927 22:51:04.071863 16909 ts_tablet_manager.cc:1918] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:04.072005 16909 log.cc:1196] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/56b1ce0fba944905bfac0227523683cd
I20260927 22:51:04.072420 16909 ts_tablet_manager.cc:1939] T 56b1ce0fba944905bfac0227523683cd P f4585d7a5c5048cea194b42a1de1c5ab: Deleting consensus metadata
I20260927 22:51:04.073698 16406 catalog_manager.cc:4841] TS f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019): tablet 56b1ce0fba944905bfac0227523683cd (table test-table [id=bea53626f30240abb7ff290c39238452]) successfully deleted
I20260927 22:51:04.183645 16908 tablet_replica.cc:330] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: stopping tablet replica
I20260927 22:51:04.183905 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 16456
W20260927 22:51:04.198067 16403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39491 (error 108)
I20260927 22:51:04.198305 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 16609
I20260927 22:51:04.200165 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.199712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46788)
0927 22:51:04.199887 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:04.200017 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:51:04.200377 16403 catalog_manager.cc:4584] TS db67e52227504cefa263cb9ed1987f97 (127.0.97.1:39491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56b1ce0fba944905bfac0227523683cd: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
I20260927 22:51:04.214709 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.199053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.0.97.2:40019)
0927 22:51:04.199235 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:04.199241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:04.199454 (+ 213us) server_negotiation.cc:407] Connection header received
0927 22:51:04.199629 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:04.199633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:04.199697 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:04.199782 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:04.201698 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:04.202575 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:04.204071 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:04.204548 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:04.204672 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:04.207812 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:04.207826 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:04.207854 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:04.207905 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:04.211825 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:04.212541 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:04.212552 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:04.212556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:04.212727 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:04.213101 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:04.213107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:04.213110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:04.213580 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:04.213932 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:04.214089 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:51:04.214324 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:51:04.215396 16732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:04.254077 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.253562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46790)
0927 22:51:04.253757 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:04.253872 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:51:04.255209 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.254804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46806)
0927 22:51:04.254946 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:04.255056 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:51:04.315833 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.315326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46822)
0927 22:51:04.315578 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:04.315667 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:51:04.316815 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.316487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46828)
0927 22:51:04.316633 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:04.316703 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 22:51:04.325843 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 16609
I20260927 22:51:04.340371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 16740
I20260927 22:51:04.352381 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.341101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.0.97.3:42009)
0927 22:51:04.341280 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:04.341285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:04.341301 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:51:04.341432 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:04.341435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:04.341507 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:04.341589 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:04.342856 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:04.343599 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:04.344759 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:04.345131 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:04.345214 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:04.347600 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:04.347614 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:04.347630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:04.347663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:04.350143 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:04.350716 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:04.350722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:04.350724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:04.350849 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:04.351129 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:04.351132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:04.351134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:04.351481 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:04.351699 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:04.351911 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:51:04.352077 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:51:04.353016 16863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:04.397166 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.396723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46834)
0927 22:51:04.396929 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:04.397018 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:51:04.398370 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:04.397958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39491 (local address 127.0.0.1:46850)
0927 22:51:04.398161 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:04.398237 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:51:04.440090 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 16740
I20260927 22:51:04.454042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 16386
I20260927 22:51:04.455233 16449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:04.568581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 16386
I20260927 22:51:24.586766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34779
--webserver_interface=127.0.97.62
--webserver_port=42911
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:24.908387 16916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:24.908674 16916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:24.908807 16916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:24.916062 16916 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:51:24.916184 16916 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:51:24.916244 16916 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:51:24.916309 16916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:24.916383 16916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:24.916478 16916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:24.916538 16916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:24.916612 16916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:24.926448 16916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:24.927742 16916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:24.929050 16916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:24.940207 16921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:24.940275 16924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:24.940277 16922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:24.940523 16916 server_base.cc:1034] running on GCE node
I20260927 22:51:24.941493 16916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:24.942896 16916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:24.944113 16916 hybrid_clock.cc:648] HybridClock initialized: now 1790549484944049 us; error 42 us; skew 500 ppm
I20260927 22:51:24.944413 16916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:24 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:24.947314 16916 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:24.948668 16916 webserver.cc:460] Webserver started at http://127.0.97.62:42911/ using document root <none> and password file <none>
I20260927 22:51:24.949316 16916 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:24.949407 16916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:24.954945 16916 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:24.958024 16931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:24.959370 16916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:51:24.959555 16916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "24f8d4caeee54e89940b5fffc77edf0c"
format_stamp: "Formatted at 2026-09-27 22:50:39 on dist-test-slave-kr25"
server_key: "32bead642361ba73cb8a8534c1e95d24"
server_key_iv: "55fca012781e3f4b0c9e7584d246405b"
server_key_version: "encryptionkey@0"
I20260927 22:51:24.960495 16916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:24.975812 16916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:24.979749 16916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:24.980003 16916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:24.997133 16916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34779
I20260927 22:51:24.997148 16982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34779 every 8 connection(s)
I20260927 22:51:24.998625 16916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:25.005687 16983 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:51:25.008164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16916
I20260927 22:51:25.009135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39491
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34779
--webserver_port=41801
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:51:25.009522 16983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Bootstrap starting.
I20260927 22:51:25.031324 16983 log.cc:824] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:25.047660 16983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:51:25.048154 16983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Bootstrap complete.
I20260927 22:51:25.057057 16983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:25.057968 16983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f8d4caeee54e89940b5fffc77edf0c, State: Initialized, Role: FOLLOWER
I20260927 22:51:25.058578 16983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:25.058799 16983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:25.058907 16983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:25.059053 16983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:51:25.062856 16983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:25.063346 16983 leader_election.cc:302] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f8d4caeee54e89940b5fffc77edf0c; no voters:
I20260927 22:51:25.063982 16983 leader_election.cc:288] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:51:25.064129 16987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:51:25.066169 16983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:25.067701 16987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [term 3 LEADER]: Becoming Leader. State: Replica: 24f8d4caeee54e89940b5fffc77edf0c, State: Running, Role: LEADER
I20260927 22:51:25.068292 16987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } }
I20260927 22:51:25.070946 16988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "24f8d4caeee54e89940b5fffc77edf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } } }
I20260927 22:51:25.071309 16988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:25.071818 16989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f8d4caeee54e89940b5fffc77edf0c. Latest consensus state: current_term: 3 leader_uuid: "24f8d4caeee54e89940b5fffc77edf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8d4caeee54e89940b5fffc77edf0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34779 } } }
I20260927 22:51:25.072125 16989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:25.081810 16993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:25.089576 16993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:25.090961 16993 catalog_manager.cc:1259] Loaded cluster ID: 0bea33be67514e8798d611dea82b50df
I20260927 22:51:25.091054 16993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:25.095134 16993 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:25.096647 16993 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Loaded TSK: 0
I20260927 22:51:25.096776 16993 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24f8d4caeee54e89940b5fffc77edf0c: Loaded TSK: 1
F20260927 22:51:25.097826 16993 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1c8b0b69ad google::LogMessage::Fail() at ??:0
@ 0x7f1c8b0b988a google::LogMessage::SendToLog() at ??:0
@ 0x7f1c8b0b64de google::LogMessage::Flush() at ??:0
@ 0x7f1c8b0b7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1c8ee1a7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1c8ee0973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1ca0ae51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1ca0ae4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1ca0b702a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1ca0b700c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1ca0bba5e9 std::function<>::operator()() at ??:0
@ 0x7f1ca0ae8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1ca0ae6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1ca0b6e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1ca0b6e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1c8dd02345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1c8dd0591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1c8dd05799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1c8dce0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1c90fba6db start_thread at ??:0
@ 0x7f1c89f5061f clone at ??:0
W20260927 22:51:25.423527 16985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:25.423807 16985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:25.423902 16985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:25.431087 16985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:25.431213 16985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:25.431329 16985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:25.441473 16985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41801
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:25.442755 16985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:25.444087 16985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:25.456544 17030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:25.456632 17033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:25.456504 17031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:25.457247 16985 server_base.cc:1034] running on GCE node
I20260927 22:51:25.457794 16985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:25.458662 16985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:25.459856 16985 hybrid_clock.cc:648] HybridClock initialized: now 1790549485459802 us; error 49 us; skew 500 ppm
I20260927 22:51:25.460196 16985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:25 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:25.463091 16985 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:25.464540 16985 webserver.cc:460] Webserver started at http://127.0.97.1:41801/ using document root <none> and password file <none>
I20260927 22:51:25.465179 16985 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:25.465296 16985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:25.470369 16985 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 22:51:25.473274 17040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:25.474611 16985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:25.474756 16985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "db67e52227504cefa263cb9ed1987f97"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "be7a5aa00173983e0a25622f27ef8324"
server_key_iv: "340a54aae8ac335579f81d995f774054"
server_key_version: "encryptionkey@0"
I20260927 22:51:25.475764 16985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:25.501984 16985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:25.505846 16985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:25.506093 16985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:25.507050 16985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:25.508719 17047 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 22:51:25.514219 16985 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 22:51:25.514359 16985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 22:51:25.514479 16985 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 22:51:25.517634 16985 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 22:51:25.517719 16985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:25.518115 17047 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Bootstrap starting.
I20260927 22:51:25.543751 17047 log.cc:824] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:25.553787 16985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39491
I20260927 22:51:25.553818 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39491 every 8 connection(s)
I20260927 22:51:25.555132 16985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:25.555540 17047 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:51:25.556169 17047 tablet_bootstrap.cc:492] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Bootstrap complete.
I20260927 22:51:25.557528 17047 ts_tablet_manager.cc:1397] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.012s
I20260927 22:51:25.559930 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.557837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:57689)
0927 22:51:25.558657 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.558935 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":357,"threads_started":1}
W20260927 22:51:25.560611 17155 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34779 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34779: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
I20260927 22:51:25.561456 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.561151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:54733)
0927 22:51:25.561288 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.561352 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:51:25.562278 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.561922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:39049)
0927 22:51:25.562068 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.562152 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:51:25.562491 17155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:51:25.564754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 16985
I20260927 22:51:25.565316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40019
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34779
--webserver_port=34275
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:51:25.564795 17047 raft_consensus.cc:348] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:25.565682 17047 raft_consensus.cc:729] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db67e52227504cefa263cb9ed1987f97, State: Initialized, Role: FOLLOWER
I20260927 22:51:25.566190 17047 consensus_queue.cc:260] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:25.567631 17047 ts_tablet_manager.cc:1428] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 22:51:25.567740 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.567308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:54301)
0927 22:51:25.567480 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.567558 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:51:25.568516 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.568223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:51253)
0927 22:51:25.568347 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.568407 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 22:51:25.879076 17159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:25.879365 17159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:25.879522 17159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:25.886790 17159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:25.886935 17159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:25.887058 17159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:25.896804 17159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34275
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:25.898319 17159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:25.899633 17159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:25.912279 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:25.912240 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:25.912317 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:25.913267 17159 server_base.cc:1034] running on GCE node
I20260927 22:51:25.913898 17159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:25.914829 17159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:25.916025 17159 hybrid_clock.cc:648] HybridClock initialized: now 1790549485915979 us; error 36 us; skew 500 ppm
I20260927 22:51:25.916326 17159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:25 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:25.919203 17159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:25.920637 17159 webserver.cc:460] Webserver started at http://127.0.97.2:34275/ using document root <none> and password file <none>
I20260927 22:51:25.921283 17159 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:25.921398 17159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:25.926498 17159 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 22:51:25.929046 17174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:25.930261 17159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:25.930405 17159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "f4585d7a5c5048cea194b42a1de1c5ab"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "b3166e8f2bb99ffe773d322a921564a0"
server_key_iv: "080d9168a44f02cbcf8adf1aecce729b"
server_key_version: "encryptionkey@0"
I20260927 22:51:25.931339 17159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:25.950739 17159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:25.954596 17159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:25.954847 17159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:25.955886 17159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:25.957195 17159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:25.957288 17159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:25.957401 17159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:25.957468 17159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:25.985536 17159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40019
I20260927 22:51:25.985596 17286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40019 every 8 connection(s)
I20260927 22:51:25.986835 17159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:25.991297 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.989514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:52089)
0927 22:51:25.990222 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.990431 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":266,"threads_started":1}
W20260927 22:51:25.991995 17287 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34779 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34779: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
I20260927 22:51:25.992792 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.992457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:45251)
0927 22:51:25.992614 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.992686 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:51:25.993533 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.993248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:52161)
0927 22:51:25.993362 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:25.993431 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 22:51:25.993736 17287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:51:25.994969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 17159
I20260927 22:51:25.995532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42009
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34779
--webserver_port=43265
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:26.304065 17290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:26.304330 17290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:26.304466 17290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:26.311390 17290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:26.311527 17290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:26.311676 17290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:26.321436 17290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43265
--tserver_master_addrs=127.0.97.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:26.322755 17290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:26.324041 17290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:26.336215 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:26.336253 17298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:26.336179 17296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:26.337258 17290 server_base.cc:1034] running on GCE node
I20260927 22:51:26.337832 17290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:26.338687 17290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:26.339880 17290 hybrid_clock.cc:648] HybridClock initialized: now 1790549486339838 us; error 30 us; skew 500 ppm
I20260927 22:51:26.340163 17290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:26 dist-test-slave-kr25 krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:26.342943 17290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:26.344291 17290 webserver.cc:460] Webserver started at http://127.0.97.3:43265/ using document root <none> and password file <none>
I20260927 22:51:26.344892 17290 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:26.344983 17290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:26.349848 17290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:51:26.352336 17305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:26.353657 17290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:26.353799 17290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "555105c180d9494f925fca4824d05d0a"
format_stamp: "Formatted at 2026-09-27 22:50:40 on dist-test-slave-kr25"
server_key: "742f4fc6a640bb1b0040d2649acbb3b7"
server_key_iv: "497d730120cc5998859af570ff80524d"
server_key_version: "encryptionkey@0"
I20260927 22:51:26.354779 17290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:26.381412 17290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:26.385263 17290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:26.385514 17290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:26.386430 17290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:26.387728 17290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:26.387811 17290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:26.387917 17290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:26.387981 17290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:26.417477 17290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42009
I20260927 22:51:26.417547 17417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42009 every 8 connection(s)
I20260927 22:51:26.418850 17290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:26.423513 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.421719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:46377)
0927 22:51:26.422445 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:26.422643 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":284,"threads_started":1}
W20260927 22:51:26.424120 17418 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34779 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34779: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
I20260927 22:51:26.424820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 17290
I20260927 22:51:26.424937 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.424638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:51593)
0927 22:51:26.424772 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:26.424838 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:51:26.425727 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.425423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:34891)
0927 22:51:26.425549 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:26.425623 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:51:26.425987 17418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:51:26.569980 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.569241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:42619)
0927 22:51:26.569677 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:26.569769 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":205,"threads_started":1}
I20260927 22:51:26.985251 17423 raft_consensus.cc:482] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:26.985561 17423 raft_consensus.cc:504] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:26.987499 17423 leader_election.cc:288] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
I20260927 22:51:26.995321 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.994321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:35983)
0927 22:51:26.994827 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:26.994998 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
Sep 27 22:51:26 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 27 22:51:26 dist-test-slave-kr25 krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:27.010013 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.987988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40019 (local address 127.0.97.1:58455)
0927 22:51:26.989361 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:26.989376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:26.989636 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:26.992612 (+ 2976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:26.992616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:26.993229 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:26.993933 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:26.993954 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.995881 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.995885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:26.997282 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:26.997293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.998175 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.998181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:26.998457 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:26.999211 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:26.999228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:27.001959 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:27.005784 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:27.005794 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:27.005810 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:27.006561 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:27.007002 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:27.007006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:27.007010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:27.007259 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:27.008206 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:27.008215 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:27.008701 (+ 486us) client_negotiation.cc:769] Sending connection context
0927 22:51:27.009019 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 22:51:27.009535 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":236,"spinlock_wait_cycles":14976,"thread_start_us":1113,"threads_started":1}
I20260927 22:51:27.010069 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.988270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58455 (local address 127.0.97.2:40019)
0927 22:51:26.989942 (+ 1672us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:26.989952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:26.989979 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:51:26.991521 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:26.991530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:26.991865 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:26.992165 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:26.994153 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.995714 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.997413 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.997993 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.998149 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:27.002213 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:27.002222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:27.002239 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:27.002288 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:27.005629 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.006682 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.006690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.006700 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.006888 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.007363 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.007368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.007371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.008060 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:27.008225 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:27.009260 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0927 22:51:27.009665 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1475,"spinlock_wait_cycles":20352,"thread_start_us":1456,"threads_started":1}
I20260927 22:51:27.010064 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.987933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47671 (local address 127.0.97.3:42009)
0927 22:51:26.989073 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:26.989087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:26.989126 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:51:26.991147 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:26.991159 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:26.991694 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:26.992048 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:26.994150 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.995758 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.997411 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.998003 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.998153 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:27.001930 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:27.001937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:27.001954 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:27.002044 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:27.005630 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.006682 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.006690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.006700 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.006888 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.007363 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.007368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.007371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.008053 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:27.008225 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:27.009260 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0927 22:51:27.009665 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":856,"threads_started":1}
I20260927 22:51:27.010027 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:26.987645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42009 (local address 127.0.97.1:47671)
0927 22:51:26.987816 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:26.987856 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:26.988361 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:26.992211 (+ 3850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:26.992220 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:26.993203 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:26.993936 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:26.993956 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.995916 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.995922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:26.997280 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:26.997292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:26.998222 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:26.998229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:26.998461 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:26.999170 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:26.999200 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:27.001647 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:27.005784 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:27.005794 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:27.005810 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:27.006547 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:27.007016 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:27.007020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:27.007023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:27.007260 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:27.008206 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:27.008215 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:27.008701 (+ 486us) client_negotiation.cc:769] Sending connection context
0927 22:51:27.009019 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 22:51:27.009662 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":370,"spinlock_wait_cycles":51968}
W20260927 22:51:27.013675 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
W20260927 22:51:27.014042 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
I20260927 22:51:27.014156 17044 leader_election.cc:302] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db67e52227504cefa263cb9ed1987f97; no voters: 555105c180d9494f925fca4824d05d0a, f4585d7a5c5048cea194b42a1de1c5ab
I20260927 22:51:27.014496 17423 raft_consensus.cc:2726] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:51:27.427240 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:27.426448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:52211)
0927 22:51:27.426896 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:27.427017 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260927 22:51:27.571502 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:27.570843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:51615)
0927 22:51:27.571228 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:27.571309 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":194,"threads_started":1}
I20260927 22:51:27.997115 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:27.996344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:49789)
0927 22:51:27.996836 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:27.996937 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
I20260927 22:51:28.428776 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:28.428095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:41641)
0927 22:51:28.428542 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:28.428626 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":243,"threads_started":1}
I20260927 22:51:28.573048 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:28.572286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:40523)
0927 22:51:28.572769 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:28.572850 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":222,"threads_started":1}
I20260927 22:51:28.821643 17433 raft_consensus.cc:482] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:28.821822 17433 raft_consensus.cc:504] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:28.822415 17433 leader_election.cc:288] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
W20260927 22:51:28.823410 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
W20260927 22:51:28.823666 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
I20260927 22:51:28.823750 17044 leader_election.cc:302] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db67e52227504cefa263cb9ed1987f97; no voters: 555105c180d9494f925fca4824d05d0a, f4585d7a5c5048cea194b42a1de1c5ab
I20260927 22:51:28.824008 17433 raft_consensus.cc:2726] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:51:28.998663 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:28.997996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:49743)
0927 22:51:28.998414 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:28.998501 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":221,"threads_started":1}
I20260927 22:51:29.430328 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:29.429630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:37263)
0927 22:51:29.430079 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:29.430172 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
I20260927 22:51:29.574438 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:29.573798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:55475)
0927 22:51:29.574200 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:29.574276 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":208,"threads_started":1}
I20260927 22:51:30.000368 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:29.999579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:47231)
0927 22:51:30.000075 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:30.000161 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
I20260927 22:51:30.431937 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:30.431175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:39061)
0927 22:51:30.431662 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:30.431748 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
I20260927 22:51:30.481345 17439 raft_consensus.cc:482] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:30.481552 17439 raft_consensus.cc:504] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:30.482193 17439 leader_election.cc:288] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
W20260927 22:51:30.483093 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
W20260927 22:51:30.483290 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
I20260927 22:51:30.483389 17044 leader_election.cc:302] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db67e52227504cefa263cb9ed1987f97; no voters: 555105c180d9494f925fca4824d05d0a, f4585d7a5c5048cea194b42a1de1c5ab
I20260927 22:51:30.483646 17439 raft_consensus.cc:2726] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:51:30.576139 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:30.575406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:44517)
0927 22:51:30.575859 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:30.575951 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":203,"threads_started":1}
I20260927 22:51:31.001987 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:31.001212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:36851)
0927 22:51:31.001653 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:31.001734 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":233,"threads_started":1}
I20260927 22:51:31.433526 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:31.432832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:35787)
0927 22:51:31.433280 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:31.433362 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":251,"threads_started":1}
I20260927 22:51:31.577627 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:31.576962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:52911)
0927 22:51:31.577373 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:31.577458 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260927 22:51:32.003660 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:32.002822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:36473)
0927 22:51:32.003286 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:32.003377 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260927 22:51:32.331944 17445 raft_consensus.cc:482] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:32.332134 17445 raft_consensus.cc:504] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:32.332752 17445 leader_election.cc:288] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
W20260927 22:51:32.333660 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
W20260927 22:51:32.333845 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
I20260927 22:51:32.333921 17044 leader_election.cc:302] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db67e52227504cefa263cb9ed1987f97; no voters: 555105c180d9494f925fca4824d05d0a, f4585d7a5c5048cea194b42a1de1c5ab
I20260927 22:51:32.334154 17445 raft_consensus.cc:2726] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:51:32.435093 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:32.434307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:46233)
0927 22:51:32.434816 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:32.434895 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":300,"threads_started":1}
I20260927 22:51:32.579197 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:32.578409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:53929)
0927 22:51:32.578883 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:32.578999 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":243,"threads_started":1}
I20260927 22:51:33.005487 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:33.004590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:56931)
0927 22:51:33.005162 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:33.005263 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":261,"threads_started":1}
I20260927 22:51:33.436960 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:33.436014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:38445)
0927 22:51:33.436615 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:33.436716 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":262,"threads_started":1}
I20260927 22:51:33.580842 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:33.580118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:38525)
0927 22:51:33.580575 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:33.580662 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":241,"threads_started":1}
I20260927 22:51:34.007131 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.006417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:35073)
0927 22:51:34.006870 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:34.006957 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":212,"threads_started":1}
I20260927 22:51:34.438562 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.437850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:39083)
0927 22:51:34.438250 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:34.438332 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":205,"threads_started":1}
I20260927 22:51:34.582404 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.581642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:52175)
0927 22:51:34.582123 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:34.582214 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":207,"threads_started":1}
I20260927 22:51:35.008769 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:35.008052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:54511)
0927 22:51:35.008527 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:35.008610 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":224,"threads_started":1}
I20260927 22:51:35.440378 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:35.439581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:41795)
0927 22:51:35.440122 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:35.440211 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":271,"threads_started":1}
I20260927 22:51:35.584035 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:35.583305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:54257)
0927 22:51:35.583783 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:35.583865 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260927 22:51:36.010403 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:36.009636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:38003)
0927 22:51:36.010120 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:36.010204 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":285,"threads_started":1}
I20260927 22:51:36.442097 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:36.441327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:48221)
0927 22:51:36.441784 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:36.441910 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260927 22:51:36.585548 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:36.584858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:46375)
0927 22:51:36.585281 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:36.585357 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":218,"threads_started":1}
I20260927 22:51:37.012095 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.011332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:42947)
0927 22:51:37.011823 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:37.011908 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":240,"threads_started":1}
I20260927 22:51:37.393136 17461 raft_consensus.cc:482] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:37.393309 17461 raft_consensus.cc:504] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db67e52227504cefa263cb9ed1987f97" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39491 } } peers { permanent_uuid: "555105c180d9494f925fca4824d05d0a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42009 } } peers { permanent_uuid: "f4585d7a5c5048cea194b42a1de1c5ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40019 } }
I20260927 22:51:37.394013 17461 leader_election.cc:288] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009), f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019)
W20260927 22:51:37.394949 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 555105c180d9494f925fca4824d05d0a (127.0.97.3:42009): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
W20260927 22:51:37.395153 17044 leader_election.cc:341] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4585d7a5c5048cea194b42a1de1c5ab (127.0.97.2:40019): Not found: Tablet not found: 56b1ce0fba944905bfac0227523683cd
I20260927 22:51:37.395298 17044 leader_election.cc:302] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db67e52227504cefa263cb9ed1987f97; no voters: 555105c180d9494f925fca4824d05d0a, f4585d7a5c5048cea194b42a1de1c5ab
I20260927 22:51:37.395615 17461 raft_consensus.cc:2726] T 56b1ce0fba944905bfac0227523683cd P db67e52227504cefa263cb9ed1987f97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:51:37.443899 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.443103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:59925)
0927 22:51:37.443641 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:37.443733 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":235,"threads_started":1}
I20260927 22:51:37.587106 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.586337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:58451)
0927 22:51:37.586769 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:37.586860 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":204,"threads_started":1}
I20260927 22:51:38.013698 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:38.013015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:57129)
0927 22:51:38.013444 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:38.013521 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260927 22:51:38.445519 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:38.444792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:33375)
0927 22:51:38.445261 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:38.445340 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260927 22:51:38.588582 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:38.587892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:56959)
0927 22:51:38.588315 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:38.588390 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":213,"threads_started":1}
I20260927 22:51:39.015547 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:39.014782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:59127)
0927 22:51:39.015237 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:39.015319 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260927 22:51:39.447322 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:39.446593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:53047)
0927 22:51:39.447080 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:39.447164 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
I20260927 22:51:39.590108 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:39.589397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:43681)
0927 22:51:39.589842 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:39.589939 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":227,"threads_started":1}
I20260927 22:51:40.017186 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:40.016413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:60887)
0927 22:51:40.016894 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:40.016981 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260927 22:51:40.449050 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:40.448259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:50245)
0927 22:51:40.448761 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:40.448853 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":268,"threads_started":1}
I20260927 22:51:40.592042 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:40.591240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.1:47819)
0927 22:51:40.591765 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:40.591840 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
I20260927 22:51:41.018981 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.018254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.2:35899)
0927 22:51:41.018732 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.018823 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
I20260927 22:51:41.431102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 16985
I20260927 22:51:41.449935 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.432324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.0.97.1:39491)
0927 22:51:41.432859 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.432866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.432894 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:51:41.432961 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.432967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.433180 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.433331 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.434694 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.435929 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.437134 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.437693 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.437809 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.444196 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.444217 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.444233 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.444293 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.447156 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.447942 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.447950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.447953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.448094 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.448429 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.448434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.448437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.448837 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.449068 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.449315 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.449596 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":266,"threads_started":1}
I20260927 22:51:41.450901 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.450178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34779 (local address 127.0.97.3:44841)
0927 22:51:41.450660 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.450756 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":269,"threads_started":1}
I20260927 22:51:41.451293 17150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:41.538671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 16985
I20260927 22:51:41.552737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 17159
I20260927 22:51:41.565858 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.553436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.97.2:40019)
0927 22:51:41.553906 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.553910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.553932 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:51:41.553984 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.553987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.554052 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.554144 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.555506 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.556298 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.557613 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.558059 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.558169 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.560747 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.560752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.560755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.560786 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.563237 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.563956 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.563961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.563964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.564086 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.564453 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.564458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.564460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.564737 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.565004 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.565273 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.565489 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":248,"threads_started":1}
I20260927 22:51:41.566589 17282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:41.647553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 17159
I20260927 22:51:41.661793 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 17290
I20260927 22:51:41.674564 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.662609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.0.97.3:42009)
0927 22:51:41.663146 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.663151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.663167 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:51:41.663218 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.663221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.663287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.663391 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.664767 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.665522 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.666785 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.667186 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.667291 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.669739 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.669744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.669748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.669779 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.672115 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.672734 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.672739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.672741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.672868 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.673206 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.673210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.673212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.673461 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.673695 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.673969 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.674188 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":277,"threads_started":1}
I20260927 22:51:41.675227 17413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:41.759806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 17290
2026-09-27T22:51:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63846 ms)
[----------] 30 tests from SecurityITest (285529 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:41 dist-test-slave-kr25 krb5kdc[17483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:41 dist-test-slave-kr25 krb5kdc[17483](info): set up 2 sockets
Sep 27 22:51:41 dist-test-slave-kr25 krb5kdc[17483](info): commencing operation
krb5kdc: starting...
W20260927 22:51:42.775280 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:42 dist-test-slave-kr25 krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:42.999423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45459
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:43.325125 17496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:43.325441 17496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:43.325578 17496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:43.334444 17496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:43.334663 17496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:43.334746 17496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:43.334836 17496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:43.334909 17496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:43.345741 17496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:43.347031 17496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:43.348345 17496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:43.359874 17501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:43.359894 17502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:43.360129 17504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:43.360567 17496 server_base.cc:1034] running on GCE node
I20260927 22:51:43.361328 17496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:43.362723 17496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:43.363950 17496 hybrid_clock.cc:648] HybridClock initialized: now 1790549503363886 us; error 32 us; skew 500 ppm
I20260927 22:51:43.364300 17496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:43 dist-test-slave-kr25 krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:43.367290 17496 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:43.368746 17496 webserver.cc:460] Webserver started at http://127.0.97.62:36225/ using document root <none> and password file <none>
I20260927 22:51:43.369419 17496 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:43.369535 17496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:43.369818 17496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:43.372735 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "b4b5ae04f7f34222b6c0b867a448b87e"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-kr25"
server_key: "a5f0f070540dc307d6f139dba4b57155"
server_key_iv: "f21f52d8d2c7a2c0decbf9160e76f96c"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.373472 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "b4b5ae04f7f34222b6c0b867a448b87e"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-kr25"
server_key: "a5f0f070540dc307d6f139dba4b57155"
server_key_iv: "f21f52d8d2c7a2c0decbf9160e76f96c"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.378337 17496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:51:43.381594 17511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.382825 17496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:51:43.382989 17496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "b4b5ae04f7f34222b6c0b867a448b87e"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-kr25"
server_key: "a5f0f070540dc307d6f139dba4b57155"
server_key_iv: "f21f52d8d2c7a2c0decbf9160e76f96c"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.383154 17496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:43.400079 17496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:43.403996 17496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:43.404256 17496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:43.421577 17496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45459
I20260927 22:51:43.421605 17562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45459 every 8 connection(s)
I20260927 22:51:43.422989 17496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:43.426620 17563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:43.431761 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 17496
I20260927 22:51:43.431955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:43.432574 388 external_mini_cluster.cc:1428] Setting key 8fdada5a7e27e92dfcdb13f18e9f5b7f
I20260927 22:51:43.437297 17563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e: Bootstrap starting.
I20260927 22:51:43.441171 17563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:43.442495 17563 log.cc:824] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:43.446094 17563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e: No bootstrap required, opened a new log
Sep 27 22:51:43 dist-test-slave-kr25 krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:43.455075 17563 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45459 } }
I20260927 22:51:43.455586 17563 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:43.455693 17563 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b5ae04f7f34222b6c0b867a448b87e, State: Initialized, Role: FOLLOWER
I20260927 22:51:43.456303 17563 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45459 } }
I20260927 22:51:43.456530 17563 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:43.456558 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.434386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36988 (local address 127.0.97.62:45459)
0927 22:51:43.435211 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.435222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.435263 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:51:43.436924 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.436957 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.437316 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.437780 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.439721 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:43.441024 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:43.443014 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:43.444138 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:43.444296 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:43.449212 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.449220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.449239 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:43.449287 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.452428 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:43.453267 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:43.453274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:43.453281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:43.453467 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:43.453850 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:43.453853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:43.453855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:43.454390 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.454638 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.455074 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.455468 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":378,"threads_started":1}
I20260927 22:51:43.456668 17563 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:43.456883 17563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:43.459659 17563 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45459 } }
I20260927 22:51:43.460090 17563 leader_election.cc:302] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4b5ae04f7f34222b6c0b867a448b87e; no voters:
I20260927 22:51:43.460596 17563 leader_election.cc:288] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:43.460727 17568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:43.461328 17568 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [term 1 LEADER]: Becoming Leader. State: Replica: b4b5ae04f7f34222b6c0b867a448b87e, State: Running, Role: LEADER
I20260927 22:51:43.461838 17568 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45459 } }
I20260927 22:51:43.462472 17563 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:43.465148 17570 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4b5ae04f7f34222b6c0b867a448b87e. Latest consensus state: current_term: 1 leader_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45459 } } }
I20260927 22:51:43.465530 17570 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:43.470453 17569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4b5ae04f7f34222b6c0b867a448b87e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45459 } } }
I20260927 22:51:43.470750 17569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:43.472404 17574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:43.476439 17574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:43.484865 17574 catalog_manager.cc:1347] Generated new cluster ID: d0603603b0854f1eb38026a3493a147c
I20260927 22:51:43.485002 17574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:43.523511 17574 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:43.524883 17574 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:43.535135 17574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e: Generated new TSK 0
I20260927 22:51:43.536170 17574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:43.602492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45459
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:43.918090 17589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:43.918354 17589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:43.918473 17589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:43.925575 17589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:43.925688 17589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:43.925844 17589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:43.935654 17589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:43.936916 17589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:43.938171 17589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:43.950104 17594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:43.950114 17597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:43.950125 17595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:43.951128 17589 server_base.cc:1034] running on GCE node
I20260927 22:51:43.951696 17589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:43.952536 17589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:43.953729 17589 hybrid_clock.cc:648] HybridClock initialized: now 1790549503953674 us; error 46 us; skew 500 ppm
I20260927 22:51:43.953994 17589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:43 dist-test-slave-kr25 krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:43.957042 17589 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:43.958405 17589 webserver.cc:460] Webserver started at http://127.0.97.1:40029/ using document root <none> and password file <none>
I20260927 22:51:43.959050 17589 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:43.959152 17589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:43.959473 17589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:43.962421 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "0e6e00a328eb471188e9c68a2c639a44"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-kr25"
server_key: "c4d8b05dce850e74824814df26fc5974"
server_key_iv: "a4b41781196954ebf894a2ef21839573"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.963244 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0e6e00a328eb471188e9c68a2c639a44"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-kr25"
server_key: "c4d8b05dce850e74824814df26fc5974"
server_key_iv: "a4b41781196954ebf894a2ef21839573"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.968564 17589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:51:43.972096 17604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.973644 17589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:51:43.973821 17589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "0e6e00a328eb471188e9c68a2c639a44"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-kr25"
server_key: "c4d8b05dce850e74824814df26fc5974"
server_key_iv: "a4b41781196954ebf894a2ef21839573"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.974021 17589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:43.989974 17589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:43.993680 17589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:43.993950 17589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:43.995216 17589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:43.996529 17589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:43.996598 17589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.996687 17589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:43.996728 17589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:44.028313 17589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40453
I20260927 22:51:44.028332 17716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40453 every 8 connection(s)
I20260927 22:51:44.029718 17589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:44.034399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 17589
I20260927 22:51:44.034590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:44.035223 388 external_mini_cluster.cc:1428] Setting key eef29a77e4af245ea8623ef50cd6735e
Sep 27 22:51:44 dist-test-slave-kr25 krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:44.051609 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.032237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34529 (local address 127.0.97.62:45459)
0927 22:51:44.032654 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.032658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.033646 (+ 988us) server_negotiation.cc:407] Connection header received
0927 22:51:44.035701 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.035706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.035791 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.035907 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.037706 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.038468 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.040643 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.041055 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.041161 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:44.045283 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.045289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.045292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:44.045330 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.048505 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:44.049272 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:44.049277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:44.049280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:44.049431 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:44.049853 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:44.049857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:44.049860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:44.050188 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.050320 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.051020 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.051234 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":206,"threads_started":1}
I20260927 22:51:44.052496 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.032746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45459 (local address 127.0.97.1:34529)
0927 22:51:44.033466 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.033503 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:44.035412 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:44.036182 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:44.036194 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:44.036702 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:44.037467 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:44.037494 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.038644 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.038650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:44.040445 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:44.040459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.041273 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.041279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:44.041540 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:44.042258 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:44.042286 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:44.045008 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:44.048663 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:44.048670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:44.048685 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:44.049136 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:44.049545 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:44.049548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:44.049550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:44.049740 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:44.050312 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:44.050318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:44.050652 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 22:51:44.050875 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 22:51:44.051284 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":296,"threads_started":1}
I20260927 22:51:44.054090 17717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45459
I20260927 22:51:44.054375 17717 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:44.055114 17717 heartbeater.cc:511] Master 127.0.97.62:45459 requested a full tablet report, sending...
I20260927 22:51:44.057255 17528 ts_manager.cc:194] Registered new tserver with Master: 0e6e00a328eb471188e9c68a2c639a44 (127.0.97.1:40453)
I20260927 22:51:44.058655 17528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34529
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:44.090512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45459
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:44.410683 17723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:44.410995 17723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:44.411131 17723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:44.418437 17723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:44.418556 17723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:44.418712 17723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:44.428486 17723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:44.429687 17723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:44.430922 17723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:44.443117 17728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:44.443145 17731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:44.443202 17729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:44.443995 17723 server_base.cc:1034] running on GCE node
I20260927 22:51:44.444518 17723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:44.445438 17723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:44.446648 17723 hybrid_clock.cc:648] HybridClock initialized: now 1790549504446597 us; error 41 us; skew 500 ppm
I20260927 22:51:44.446962 17723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:44 dist-test-slave-kr25 krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:44.450122 17723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:44.451674 17723 webserver.cc:460] Webserver started at http://127.0.97.2:43189/ using document root <none> and password file <none>
I20260927 22:51:44.452330 17723 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:44.452425 17723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:44.452677 17723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:44.455608 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "6ea2e52930724935a943411f9ce7144b"
format_stamp: "Formatted at 2026-09-27 22:51:44 on dist-test-slave-kr25"
server_key: "5e1e2f0463a90589fe95a39d10c6513c"
server_key_iv: "217e58bd8527f4823820f602eff7dd8f"
server_key_version: "encryptionkey@0"
I20260927 22:51:44.456333 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6ea2e52930724935a943411f9ce7144b"
format_stamp: "Formatted at 2026-09-27 22:51:44 on dist-test-slave-kr25"
server_key: "5e1e2f0463a90589fe95a39d10c6513c"
server_key_iv: "217e58bd8527f4823820f602eff7dd8f"
server_key_version: "encryptionkey@0"
I20260927 22:51:44.461298 17723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260927 22:51:44.464792 17738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:44.466050 17723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:51:44.466176 17723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "6ea2e52930724935a943411f9ce7144b"
format_stamp: "Formatted at 2026-09-27 22:51:44 on dist-test-slave-kr25"
server_key: "5e1e2f0463a90589fe95a39d10c6513c"
server_key_iv: "217e58bd8527f4823820f602eff7dd8f"
server_key_version: "encryptionkey@0"
I20260927 22:51:44.466341 17723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:44.486244 17723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:44.490553 17723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:44.490795 17723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:44.491828 17723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:44.493185 17723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:44.493257 17723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:44.493350 17723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:44.493393 17723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:44.523007 17723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32801
I20260927 22:51:44.523062 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32801 every 8 connection(s)
I20260927 22:51:44.524235 17723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:44.533128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 17723
I20260927 22:51:44.533324 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:44.533871 388 external_mini_cluster.cc:1428] Setting key 7434052e49832fa3d4bf89b73aec7b16
Sep 27 22:51:44 dist-test-slave-kr25 krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:44.544214 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.526705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39539 (local address 127.0.97.62:45459)
0927 22:51:44.526926 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.526932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.528228 (+ 1296us) server_negotiation.cc:407] Connection header received
0927 22:51:44.530026 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.530031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.530114 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.530207 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.531695 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.532397 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.534468 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.535067 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.535203 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:44.538970 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.538975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.538978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:44.539012 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.541374 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:44.542057 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:44.542061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:44.542063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:44.542184 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:44.542589 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:44.542592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:44.542594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:44.542851 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.542982 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.543665 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.543886 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:51:44.545164 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.527271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45459 (local address 127.0.97.2:39539)
0927 22:51:44.528052 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.528092 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:44.529771 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:44.530401 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:44.530410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:44.530930 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:44.531542 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:44.531561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.532507 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.532511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:44.534331 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:44.534344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:44.535306 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:44.535314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:44.535610 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:44.536321 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:44.536362 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:44.538745 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:44.541506 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:44.541515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:44.541531 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:44.541933 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:44.542289 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:44.542294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:44.542298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:44.542472 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:44.542977 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:44.542985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:44.543290 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 22:51:44.543556 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 22:51:44.544058 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":335,"threads_started":1}
I20260927 22:51:44.546511 17851 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45459
I20260927 22:51:44.546816 17851 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:44.547554 17851 heartbeater.cc:511] Master 127.0.97.62:45459 requested a full tablet report, sending...
I20260927 22:51:44.548902 17528 ts_manager.cc:194] Registered new tserver with Master: 6ea2e52930724935a943411f9ce7144b (127.0.97.2:32801)
I20260927 22:51:44.549674 17528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39539
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:44.582782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45459
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:44.893870 17856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:44.894155 17856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:44.894250 17856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:44.901703 17856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:44.901832 17856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:44.901947 17856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:44.911753 17856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:44.912955 17856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:44.914161 17856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:44.927239 17862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:44.927351 17861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:44.927312 17864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:44.928458 17856 server_base.cc:1034] running on GCE node
I20260927 22:51:44.929128 17856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:44.930137 17856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:44.931355 17856 hybrid_clock.cc:648] HybridClock initialized: now 1790549504931295 us; error 49 us; skew 500 ppm
I20260927 22:51:44.931704 17856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:44 dist-test-slave-kr25 krb5kdc[17483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:44.934983 17856 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:44.936507 17856 webserver.cc:460] Webserver started at http://127.0.97.3:45211/ using document root <none> and password file <none>
I20260927 22:51:44.937199 17856 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:44.937317 17856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:44.937609 17856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:44.940541 17856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "0d346471f15a477ea7d42a13ae03a4f3"
format_stamp: "Formatted at 2026-09-27 22:51:44 on dist-test-slave-kr25"
server_key: "ac078e930cb5fb355c260a6b6494341f"
server_key_iv: "7a22fa45295f1fc7cfe09f48be55aa77"
server_key_version: "encryptionkey@0"
I20260927 22:51:44.941313 17856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0d346471f15a477ea7d42a13ae03a4f3"
format_stamp: "Formatted at 2026-09-27 22:51:44 on dist-test-slave-kr25"
server_key: "ac078e930cb5fb355c260a6b6494341f"
server_key_iv: "7a22fa45295f1fc7cfe09f48be55aa77"
server_key_version: "encryptionkey@0"
I20260927 22:51:44.946337 17856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:51:44.949750 17871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:44.951069 17856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:51:44.951223 17856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "0d346471f15a477ea7d42a13ae03a4f3"
format_stamp: "Formatted at 2026-09-27 22:51:44 on dist-test-slave-kr25"
server_key: "ac078e930cb5fb355c260a6b6494341f"
server_key_iv: "7a22fa45295f1fc7cfe09f48be55aa77"
server_key_version: "encryptionkey@0"
I20260927 22:51:44.951395 17856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:44.982409 17856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:44.986790 17856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:44.987056 17856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:44.988081 17856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:44.989390 17856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:44.989466 17856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:44.989583 17856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:44.989643 17856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:45.019385 17856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34813
I20260927 22:51:45.019459 17983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34813 every 8 connection(s)
I20260927 22:51:45.020675 17856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:45.024698 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 17856
I20260927 22:51:45.024873 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:45.025283 388 external_mini_cluster.cc:1428] Setting key 862da4b9269fd11f760c20414ebe1e35
Sep 27 22:51:45 dist-test-slave-kr25 krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:45.040524 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.023004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37929 (local address 127.0.97.62:45459)
0927 22:51:45.023239 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.023245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.024542 (+ 1297us) server_negotiation.cc:407] Connection header received
0927 22:51:45.026648 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.026653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.026737 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.026826 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:45.028608 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.029354 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.030867 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.031240 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.031344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.034711 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:45.034715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:45.034718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:45.034751 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:45.037351 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.038043 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.038047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.038050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.038173 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.038618 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.038622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.038624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.038949 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:45.039107 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.040006 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.040191 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:51:45.041445 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.023574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45459 (local address 127.0.97.3:37929)
0927 22:51:45.024330 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.024375 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.026348 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.027117 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.027128 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.027743 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:45.028443 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.028463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.029506 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.029510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.030721 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.030733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.031493 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.031501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.031756 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.032386 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:45.032409 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:45.034501 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:45.037499 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:45.037508 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:45.037524 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:45.037914 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:45.038281 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:45.038286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:45.038289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:45.038501 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:45.039092 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:45.039100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:45.039518 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 22:51:45.039827 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 22:51:45.040317 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":307,"threads_started":1}
I20260927 22:51:45.042676 17984 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45459
I20260927 22:51:45.042958 17984 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:45.043689 17984 heartbeater.cc:511] Master 127.0.97.62:45459 requested a full tablet report, sending...
I20260927 22:51:45.045006 17528 ts_manager.cc:194] Registered new tserver with Master: 0d346471f15a477ea7d42a13ae03a4f3 (127.0.97.3:34813)
I20260927 22:51:45.045807 17528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37929
I20260927 22:51:45.053718 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:45.061630 17717 heartbeater.cc:503] Master 127.0.97.62:45459 was elected leader, sending a full tablet report...
I20260927 22:51:45.070488 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.058137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.0.97.62:45459)
0927 22:51:45.058363 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.058369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.058662 (+ 293us) server_negotiation.cc:407] Connection header received
0927 22:51:45.058824 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.058829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.058899 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.059005 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:45.060505 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.061207 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.062723 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.063270 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.063403 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.065537 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:45.065542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:45.065545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:45.065577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:45.068197 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.068816 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.068821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.068825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.068944 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.069223 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.069226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.069228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.069572 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:45.069766 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.070067 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.070221 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:51:45.074268 17528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:45.081000 17528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:45.101325 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.092975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.0.97.2:32801)
0927 22:51:45.094266 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.094275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.094307 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:51:45.094418 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.094427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.094717 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.095724 (+ 1007us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:45.095945 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.097248 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.099248 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.100216 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.100332 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.100538 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.100842 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.101014 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":1051,"threads_started":1}
I20260927 22:51:45.101385 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.092773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32801 (local address 127.0.0.1:32858)
0927 22:51:45.093612 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.093655 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.093845 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.094945 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.094953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.094986 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:45.095507 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.095523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.097368 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.097373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.099051 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.099065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.100424 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.100430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.100630 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.100667 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:51:45.100781 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:51:45.100972 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":445,"threads_started":1}
I20260927 22:51:45.103763 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.094862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40453 (local address 127.0.0.1:46452)
0927 22:51:45.095417 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.095454 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.095570 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.096543 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.096547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.096570 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:45.097106 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.097120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.098991 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.098999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.101145 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.101159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.102935 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.102941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.103089 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.103124 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:51:45.103187 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 22:51:45.103315 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":2,"thread_start_us":259,"threads_started":1}
I20260927 22:51:45.104365 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.092950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34813 (local address 127.0.0.1:41854)
0927 22:51:45.094566 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.094583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.094738 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.096487 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.096492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.096517 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:45.097050 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.097062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.099327 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.099332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.101497 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.101510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.103590 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.103597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.103787 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.103819 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:51:45.103904 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:51:45.104025 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"mutex_wait_us":60,"thread_start_us":1530,"threads_started":1}
I20260927 22:51:45.104794 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.095091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.97.1:40453)
0927 22:51:45.096102 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.096109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.096129 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:51:45.096201 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.096206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.096369 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.097039 (+ 670us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:45.097253 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.098817 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.101319 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.102721 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.104076 (+ 1355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.104238 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.104290 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.104398 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":702,"threads_started":1}
I20260927 22:51:45.105046 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.093598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.0.97.3:34813)
0927 22:51:45.094603 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.094611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.094661 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:51:45.096045 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.096053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.096275 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.096477 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:45.097278 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.099164 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.101662 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.103354 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.103503 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.103726 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.104588 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.104746 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":687,"threads_started":1}
I20260927 22:51:45.105590 17786 tablet_service.cc:1467] Processing CreateTablet for tablet 0acab576154d4f4e98771086ef8f4817 (DEFAULT_TABLE table=test-table [id=5b4da25382594d1d8234f4d838cfea01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:45.107054 17786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acab576154d4f4e98771086ef8f4817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:45.108215 17919 tablet_service.cc:1467] Processing CreateTablet for tablet 0acab576154d4f4e98771086ef8f4817 (DEFAULT_TABLE table=test-table [id=5b4da25382594d1d8234f4d838cfea01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:45.108327 17652 tablet_service.cc:1467] Processing CreateTablet for tablet 0acab576154d4f4e98771086ef8f4817 (DEFAULT_TABLE table=test-table [id=5b4da25382594d1d8234f4d838cfea01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:45.109330 17919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acab576154d4f4e98771086ef8f4817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:45.109622 17652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acab576154d4f4e98771086ef8f4817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:45.119714 18002 tablet_bootstrap.cc:492] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Bootstrap starting.
I20260927 22:51:45.121687 18003 tablet_bootstrap.cc:492] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Bootstrap starting.
I20260927 22:51:45.123009 18004 tablet_bootstrap.cc:492] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Bootstrap starting.
I20260927 22:51:45.124348 18002 tablet_bootstrap.cc:654] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:45.125401 18003 tablet_bootstrap.cc:654] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:45.125939 18004 tablet_bootstrap.cc:654] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:45.126098 18002 log.cc:824] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:45.126623 18003 log.cc:824] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:45.126991 18004 log.cc:824] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:45.129338 18002 tablet_bootstrap.cc:492] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: No bootstrap required, opened a new log
I20260927 22:51:45.129593 18002 ts_tablet_manager.cc:1397] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 22:51:45.129659 18004 tablet_bootstrap.cc:492] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: No bootstrap required, opened a new log
I20260927 22:51:45.130002 18004 ts_tablet_manager.cc:1397] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260927 22:51:45.130386 18003 tablet_bootstrap.cc:492] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: No bootstrap required, opened a new log
I20260927 22:51:45.130656 18003 ts_tablet_manager.cc:1397] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:51:45.137334 18002 raft_consensus.cc:348] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.137676 18002 raft_consensus.cc:374] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:45.137785 18002 raft_consensus.cc:729] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d346471f15a477ea7d42a13ae03a4f3, State: Initialized, Role: FOLLOWER
I20260927 22:51:45.138330 18002 consensus_queue.cc:260] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.139313 17984 heartbeater.cc:503] Master 127.0.97.62:45459 was elected leader, sending a full tablet report...
I20260927 22:51:45.140195 18004 raft_consensus.cc:348] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.140668 18004 raft_consensus.cc:374] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:45.140795 18004 raft_consensus.cc:729] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6e00a328eb471188e9c68a2c639a44, State: Initialized, Role: FOLLOWER
I20260927 22:51:45.140628 18003 raft_consensus.cc:348] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.141060 18003 raft_consensus.cc:374] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:45.141173 18003 raft_consensus.cc:729] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea2e52930724935a943411f9ce7144b, State: Initialized, Role: FOLLOWER
I20260927 22:51:45.141383 18004 consensus_queue.cc:260] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.141719 18003 consensus_queue.cc:260] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.143060 18004 ts_tablet_manager.cc:1428] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260927 22:51:45.143852 18002 ts_tablet_manager.cc:1428] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Time spent starting tablet: real 0.014s user 0.005s sys 0.006s
I20260927 22:51:45.144119 17851 heartbeater.cc:503] Master 127.0.97.62:45459 was elected leader, sending a full tablet report...
I20260927 22:51:45.144788 18003 ts_tablet_manager.cc:1428] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 22:51:45.229290 18008 raft_consensus.cc:482] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:45.229597 18008 raft_consensus.cc:504] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.231324 18008 leader_election.cc:288] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea2e52930724935a943411f9ce7144b (127.0.97.2:32801), 0e6e00a328eb471188e9c68a2c639a44 (127.0.97.1:40453)
I20260927 22:51:45.236831 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.231683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32801 (local address 127.0.97.3:56517)
0927 22:51:45.231896 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.231915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.232052 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.232420 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.232423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.232445 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:45.232637 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.232648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.233639 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.233644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.235203 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.235221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.236320 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.236326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.236472 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.236495 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:51:45.236561 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 22:51:45.236644 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:51:45.237006 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.231690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56517 (local address 127.0.97.2:32801)
0927 22:51:45.231884 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.231890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.231989 (+ 99us) server_negotiation.cc:407] Connection header received
0927 22:51:45.232214 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.232219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.232299 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.232398 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:45.232783 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.233506 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.235372 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.236165 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.236265 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.236337 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.236618 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.236752 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:45.238139 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.231807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34171 (local address 127.0.97.1:40453)
0927 22:51:45.232001 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.232006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.233042 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 22:51:45.233174 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.233190 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.233247 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.233327 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:45.233893 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.234587 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.236398 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.237394 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.237505 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.237573 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.237877 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.237961 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:51:45.237944 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acab576154d4f4e98771086ef8f4817" candidate_uuid: "0d346471f15a477ea7d42a13ae03a4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea2e52930724935a943411f9ce7144b" is_pre_election: true
I20260927 22:51:45.238266 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.232234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40453 (local address 127.0.97.3:34171)
0927 22:51:45.232941 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.232954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.233055 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.233362 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.233366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.233389 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:45.233701 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.233711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.234741 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.234745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.236237 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.236251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.237567 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.237573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.237734 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.237764 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:51:45.237855 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:51:45.237967 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":412,"threads_started":1}
I20260927 22:51:45.238481 17806 raft_consensus.cc:2454] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d346471f15a477ea7d42a13ae03a4f3 in term 0.
I20260927 22:51:45.239281 17874 leader_election.cc:302] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d346471f15a477ea7d42a13ae03a4f3, 6ea2e52930724935a943411f9ce7144b; no voters:
I20260927 22:51:45.239332 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acab576154d4f4e98771086ef8f4817" candidate_uuid: "0d346471f15a477ea7d42a13ae03a4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e6e00a328eb471188e9c68a2c639a44" is_pre_election: true
I20260927 22:51:45.239709 18008 raft_consensus.cc:2781] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:45.239840 18008 raft_consensus.cc:482] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:45.239835 17672 raft_consensus.cc:2454] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d346471f15a477ea7d42a13ae03a4f3 in term 0.
I20260927 22:51:45.239960 18008 raft_consensus.cc:3037] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:45.242183 18008 raft_consensus.cc:504] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.242717 18008 leader_election.cc:288] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea2e52930724935a943411f9ce7144b (127.0.97.2:32801), 0e6e00a328eb471188e9c68a2c639a44 (127.0.97.1:40453)
I20260927 22:51:45.243175 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acab576154d4f4e98771086ef8f4817" candidate_uuid: "0d346471f15a477ea7d42a13ae03a4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e6e00a328eb471188e9c68a2c639a44"
I20260927 22:51:45.243218 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acab576154d4f4e98771086ef8f4817" candidate_uuid: "0d346471f15a477ea7d42a13ae03a4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea2e52930724935a943411f9ce7144b"
I20260927 22:51:45.243341 17672 raft_consensus.cc:3037] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:45.243387 17806 raft_consensus.cc:3037] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:45.245432 17672 raft_consensus.cc:2454] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d346471f15a477ea7d42a13ae03a4f3 in term 1.
I20260927 22:51:45.245545 17806 raft_consensus.cc:2454] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d346471f15a477ea7d42a13ae03a4f3 in term 1.
I20260927 22:51:45.245810 17873 leader_election.cc:302] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d346471f15a477ea7d42a13ae03a4f3, 0e6e00a328eb471188e9c68a2c639a44; no voters:
I20260927 22:51:45.246074 18008 raft_consensus.cc:2781] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:45.246490 18008 raft_consensus.cc:686] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0d346471f15a477ea7d42a13ae03a4f3, State: Running, Role: LEADER
I20260927 22:51:45.247000 18008 consensus_queue.cc:237] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } }
I20260927 22:51:45.251776 17526 catalog_manager.cc:5495] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d346471f15a477ea7d42a13ae03a4f3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "0d346471f15a477ea7d42a13ae03a4f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d346471f15a477ea7d42a13ae03a4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 } health_report { overall_health: UNKNOWN } } }
W20260927 22:51:45.272195 17985 tablet.cc:2369] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:45.276075 17852 tablet.cc:2369] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:45.282011 17718 tablet.cc:2369] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:45.296937 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.291575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.0.97.3:34813)
0927 22:51:45.291774 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.291779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.291857 (+ 78us) server_negotiation.cc:407] Connection header received
0927 22:51:45.292093 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.292097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.292173 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.292300 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:45.292670 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.293379 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.294834 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.295252 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.295367 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.295867 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:45.296389 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:45.296537 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.296611 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.296690 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:51:45.304802 17806 raft_consensus.cc:1261] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 1 FOLLOWER]: Refusing update from remote peer 0d346471f15a477ea7d42a13ae03a4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:45.304802 17672 raft_consensus.cc:1261] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 1 FOLLOWER]: Refusing update from remote peer 0d346471f15a477ea7d42a13ae03a4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:45.305832 18012 consensus_queue.cc:1035] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea2e52930724935a943411f9ce7144b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:45.306149 18008 consensus_queue.cc:1035] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e6e00a328eb471188e9c68a2c639a44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:45.316702 18015 mvcc.cc:205] Tried to move back new op lower bound from 7334090773718081536 to 7334090773495963648. Current Snapshot: MvccSnapshot[applied={T|T < 7334090773718081536}]
I20260927 22:51:45.329982 17526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:45.330308 17526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:45.333954 17526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4b5ae04f7f34222b6c0b867a448b87e: Sending DeleteTablet for 3 replicas of tablet 0acab576154d4f4e98771086ef8f4817
I20260927 22:51:45.335111 17786 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acab576154d4f4e98771086ef8f4817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260927 22:51:45.335270 17652 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acab576154d4f4e98771086ef8f4817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46452
I20260927 22:51:45.335491 17919 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acab576154d4f4e98771086ef8f4817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41854
I20260927 22:51:45.337335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 17589
I20260927 22:51:45.348040 18027 tablet_replica.cc:330] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: stopping tablet replica
I20260927 22:51:45.348500 18027 raft_consensus.cc:2229] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:45.349014 18027 raft_consensus.cc:2258] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:45.351234 18027 ts_tablet_manager.cc:1905] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:45.358214 18027 ts_tablet_manager.cc:1918] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:45.358422 18027 log.cc:1196] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/0acab576154d4f4e98771086ef8f4817
I20260927 22:51:45.358956 18027 ts_tablet_manager.cc:1939] T 0acab576154d4f4e98771086ef8f4817 P 6ea2e52930724935a943411f9ce7144b: Deleting consensus metadata
I20260927 22:51:45.360949 17514 catalog_manager.cc:4841] TS 6ea2e52930724935a943411f9ce7144b (127.0.97.2:32801): tablet 0acab576154d4f4e98771086ef8f4817 (table test-table [id=5b4da25382594d1d8234f4d838cfea01]) successfully deleted
I20260927 22:51:45.365167 18028 tablet_replica.cc:330] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: stopping tablet replica
I20260927 22:51:45.365720 18028 raft_consensus.cc:2229] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:45.366437 18028 raft_consensus.cc:2258] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:45.367537 18029 tablet_replica.cc:330] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: stopping tablet replica
I20260927 22:51:45.368291 18029 raft_consensus.cc:2229] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:45.369019 18029 raft_consensus.cc:2258] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:45.369539 18028 ts_tablet_manager.cc:1905] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:45.371335 18029 ts_tablet_manager.cc:1905] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:51:45 dist-test-slave-kr25 krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:45.376688 18028 ts_tablet_manager.cc:1918] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:45.376971 18028 log.cc:1196] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/0acab576154d4f4e98771086ef8f4817
I20260927 22:51:45.377651 18028 ts_tablet_manager.cc:1939] T 0acab576154d4f4e98771086ef8f4817 P 0e6e00a328eb471188e9c68a2c639a44: Deleting consensus metadata
I20260927 22:51:45.378535 18029 ts_tablet_manager.cc:1918] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:45.378734 18029 log.cc:1196] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/0acab576154d4f4e98771086ef8f4817
I20260927 22:51:45.379281 18029 ts_tablet_manager.cc:1939] T 0acab576154d4f4e98771086ef8f4817 P 0d346471f15a477ea7d42a13ae03a4f3: Deleting consensus metadata
I20260927 22:51:45.379621 17513 catalog_manager.cc:4841] TS 0e6e00a328eb471188e9c68a2c639a44 (127.0.97.1:40453): tablet 0acab576154d4f4e98771086ef8f4817 (table test-table [id=5b4da25382594d1d8234f4d838cfea01]) successfully deleted
I20260927 22:51:45.380927 17513 catalog_manager.cc:4841] TS 0d346471f15a477ea7d42a13ae03a4f3 (127.0.97.3:34813): tablet 0acab576154d4f4e98771086ef8f4817 (table test-table [id=5b4da25382594d1d8234f4d838cfea01]) successfully deleted
I20260927 22:51:45.382702 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.338661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.97.1:40453)
0927 22:51:45.338839 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.338845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.364379 (+ 25534us) server_negotiation.cc:407] Connection header received
0927 22:51:45.364572 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.364578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.364693 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.364804 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:45.366669 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.367786 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.369629 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.370229 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.370328 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.375679 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:45.375693 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:45.375710 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:45.375758 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:45.379614 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.380511 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.380520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.380524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.380674 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.381143 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.381147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.381151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.381568 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:45.381863 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.382119 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.382341 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:45.383363 17712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:45.472488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 17589
I20260927 22:51:45.487577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 17723
Sep 27 22:51:45 dist-test-slave-kr25 krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:45.502731 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.488402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.0.97.2:32801)
0927 22:51:45.488595 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.488601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.488621 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:51:45.488684 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.488690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.488777 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.488874 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:45.490146 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.490875 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.492294 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.492680 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.492800 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.497209 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:45.497225 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:45.497241 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:45.497279 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:45.500106 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.500762 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.500770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.500772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.500889 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.501251 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.501255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.501257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.501630 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:45.501903 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.502208 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.502410 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:51:45.503381 17846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:45.589659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 17723
I20260927 22:51:45.603515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 17856
Sep 27 22:51:45 dist-test-slave-kr25 krb5kdc[17483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:45.620816 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.604262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.0.97.3:34813)
0927 22:51:45.604473 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.604478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.604499 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:45.604748 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.604754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.604842 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.604962 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:45.606116 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.607339 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.608768 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.609475 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.609614 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.614072 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:45.614087 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:45.614108 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:45.614158 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:45.618001 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.618664 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.618673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.618676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.618846 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:45.619180 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:45.619185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:45.619189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:45.619682 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:45.620006 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.620166 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.620396 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:51:45.621630 17979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:45.739910 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 17856
I20260927 22:51:45.755264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 17496
I20260927 22:51:45.756453 17558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:45.864050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 17496
2026-09-27T22:51:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4105 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T22:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:45Z Disabled control of system clock
I20260927 22:51:45.931284 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35445
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:51:46.245569 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:46.245870 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:46.245991 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:46.253396 18038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:46.253513 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:46.253571 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:46.253655 18038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:46.253710 18038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:46.263093 18038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:46.264473 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:46.266186 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:46.280081 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:46.280264 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:46.280272 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:46.281452 18038 server_base.cc:1034] running on GCE node
I20260927 22:51:46.282219 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:46.283620 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:46.284832 18038 hybrid_clock.cc:648] HybridClock initialized: now 1790549506284780 us; error 43 us; skew 500 ppm
I20260927 22:51:46.285172 18038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:46.287518 18038 webserver.cc:460] Webserver started at http://127.0.97.62:34475/ using document root <none> and password file <none>
I20260927 22:51:46.288218 18038 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:46.288319 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:46.288622 18038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:46.291656 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "72caea45cab444c49acf74787edec685"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-kr25"
server_key: "7a457835c0dfaa87618e9bd4a9ca5be2"
server_key_iv: "57902a9ff9c9d9385f68a9c0643e63a1"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.292443 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "72caea45cab444c49acf74787edec685"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-kr25"
server_key: "7a457835c0dfaa87618e9bd4a9ca5be2"
server_key_iv: "57902a9ff9c9d9385f68a9c0643e63a1"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.297430 18038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:51:46.300783 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:46.302150 18038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:51:46.302314 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "72caea45cab444c49acf74787edec685"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-kr25"
server_key: "7a457835c0dfaa87618e9bd4a9ca5be2"
server_key_iv: "57902a9ff9c9d9385f68a9c0643e63a1"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.302498 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:46.324254 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:46.325084 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:46.325350 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:46.342305 18038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35445
I20260927 22:51:46.342314 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35445 every 8 connection(s)
I20260927 22:51:46.343597 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:46.347621 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:46.352270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18038
I20260927 22:51:46.352451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:46.352883 388 external_mini_cluster.cc:1428] Setting key 506f521feaf580ad4ba4b1fe83e071c8
I20260927 22:51:46.358384 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685: Bootstrap starting.
I20260927 22:51:46.361817 18104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:46.362907 18104 log.cc:824] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:46.365948 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685: No bootstrap required, opened a new log
I20260927 22:51:46.366465 18107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:46.354693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.0.97.62:35445)
0927 22:51:46.355729 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:46.355750 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:46.355792 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:51:46.357464 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:46.357486 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:46.357913 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:46.358218 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:46.358799 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.360158 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.362007 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.362737 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.362910 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:46.364168 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:46.364178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:46.364196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:46.364745 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:46.364820 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:46.364842 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:46.364955 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:46.365189 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:51:46.365490 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260927 22:51:46.373896 18104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72caea45cab444c49acf74787edec685" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35445 } }
I20260927 22:51:46.374256 18104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:46.374361 18104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72caea45cab444c49acf74787edec685, State: Initialized, Role: FOLLOWER
I20260927 22:51:46.374926 18104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72caea45cab444c49acf74787edec685" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35445 } }
I20260927 22:51:46.375138 18104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:46.375269 18104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:46.375427 18104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:46.377578 18104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72caea45cab444c49acf74787edec685" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35445 } }
I20260927 22:51:46.378065 18104 leader_election.cc:302] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72caea45cab444c49acf74787edec685; no voters:
I20260927 22:51:46.378654 18104 leader_election.cc:288] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:46.378774 18109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:46.379618 18109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [term 1 LEADER]: Becoming Leader. State: Replica: 72caea45cab444c49acf74787edec685, State: Running, Role: LEADER
I20260927 22:51:46.380376 18109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72caea45cab444c49acf74787edec685" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35445 } }
I20260927 22:51:46.380970 18104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:46.383376 18110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72caea45cab444c49acf74787edec685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72caea45cab444c49acf74787edec685" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35445 } } }
I20260927 22:51:46.383709 18110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:46.384479 18116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:46.384336 18111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72caea45cab444c49acf74787edec685. Latest consensus state: current_term: 1 leader_uuid: "72caea45cab444c49acf74787edec685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72caea45cab444c49acf74787edec685" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35445 } } }
I20260927 22:51:46.384609 18111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:46.389765 18116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:46.399880 18116 catalog_manager.cc:1347] Generated new cluster ID: 6b74e4de598d4795b5d474c5a561d3ee
I20260927 22:51:46.400000 18116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:46.426055 18116 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:46.428056 18116 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:46.442058 18116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685: Generated new TSK 0
I20260927 22:51:46.442857 18116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:51:46.456115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35445
--builtin_ntp_servers=127.0.97.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:51:46.794744 18128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:46.795001 18128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:46.795082 18128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:46.802125 18128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:46.802212 18128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:46.802312 18128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:46.811259 18128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:46.812616 18128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:46.814263 18128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:46.826625 18134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:46.826676 18133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:46.826673 18136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:46.827811 18128 server_base.cc:1034] running on GCE node
I20260927 22:51:46.828359 18128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:46.829255 18128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:46.830464 18128 hybrid_clock.cc:648] HybridClock initialized: now 1790549506830416 us; error 44 us; skew 500 ppm
I20260927 22:51:46.830752 18128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:46.836015 18128 webserver.cc:460] Webserver started at http://127.0.97.1:40251/ using document root <none> and password file <none>
I20260927 22:51:46.836670 18128 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:46.836774 18128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:46.837069 18128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:46.839802 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "bf697025d847465ca768af4ae68d0c8a"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-kr25"
server_key: "59decbe0d210aaf78925e2a60995f27a"
server_key_iv: "10b1ad4e0dc976010ce73fc5d6dfc34c"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.840559 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bf697025d847465ca768af4ae68d0c8a"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-kr25"
server_key: "59decbe0d210aaf78925e2a60995f27a"
server_key_iv: "10b1ad4e0dc976010ce73fc5d6dfc34c"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.845628 18128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:51:46.849078 18142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:46.850459 18128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:46.850607 18128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "bf697025d847465ca768af4ae68d0c8a"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-kr25"
server_key: "59decbe0d210aaf78925e2a60995f27a"
server_key_iv: "10b1ad4e0dc976010ce73fc5d6dfc34c"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.850808 18128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:46.863947 18128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:46.864687 18128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:46.864962 18128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:46.865924 18128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:46.867326 18128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:46.867417 18128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:46.867559 18128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:46.867635 18128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:46.896829 18128 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46675
I20260927 22:51:46.896884 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46675 every 8 connection(s)
I20260927 22:51:46.898095 18128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:46.901706 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18128
I20260927 22:51:46.901880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:46.902400 388 external_mini_cluster.cc:1428] Setting key 73f4e1caf83a80dda30fc88c23bfd850
I20260927 22:51:46.905213 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35445
--builtin_ntp_servers=127.0.97.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:51:46.911103 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:46.900538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39531 (local address 127.0.97.62:35445)
0927 22:51:46.901005 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:46.901011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:46.902337 (+ 1326us) server_negotiation.cc:407] Connection header received
0927 22:51:46.904276 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:46.904280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:46.904340 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:46.904435 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:46.905787 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.906519 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.908520 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.908907 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.909002 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:46.910378 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:46.910383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:46.910386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:46.910404 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:46.910455 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:46.910468 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:46.910543 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:46.910763 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 22:51:46.910842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260927 22:51:46.912366 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:46.901133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35445 (local address 127.0.97.1:39531)
0927 22:51:46.902179 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:46.902222 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:46.904033 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:46.904704 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:46.904715 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:46.904826 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:46.905618 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:46.905641 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.906653 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.906658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:46.908393 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:46.908406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.909106 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.909112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:46.909380 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:46.910053 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:46.910056 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:46.910104 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:46.910197 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:46.910570 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:46.910580 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:46.910587 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:51:46.910753 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:51:46.911113 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":310,"threads_started":1}
I20260927 22:51:46.913950 18255 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35445
I20260927 22:51:46.914218 18255 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:46.914961 18255 heartbeater.cc:511] Master 127.0.97.62:35445 requested a full tablet report, sending...
I20260927 22:51:46.917094 18069 ts_manager.cc:194] Registered new tserver with Master: bf697025d847465ca768af4ae68d0c8a (127.0.97.1:46675)
I20260927 22:51:46.918972 18069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39531
W20260927 22:51:47.219300 18259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:47.219676 18259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:47.219820 18259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:47.226904 18259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:47.227006 18259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:47.227150 18259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:47.235927 18259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:47.237102 18259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:47.238758 18259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:47.250799 18265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:47.250825 18267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:47.251140 18264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:47.251595 18259 server_base.cc:1034] running on GCE node
I20260927 22:51:47.252123 18259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:47.253019 18259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:47.254215 18259 hybrid_clock.cc:648] HybridClock initialized: now 1790549507254156 us; error 44 us; skew 500 ppm
I20260927 22:51:47.254516 18259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:47.256606 18259 webserver.cc:460] Webserver started at http://127.0.97.2:32893/ using document root <none> and password file <none>
I20260927 22:51:47.257241 18259 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:47.257323 18259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:47.257606 18259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:47.260406 18259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "6e77322e29984269b2f2391dbd638fe8"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-kr25"
server_key: "f86263137e8ff51eb11802a98d7647cb"
server_key_iv: "d5251bca63a71a9a7ec5959b9795269d"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.261122 18259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6e77322e29984269b2f2391dbd638fe8"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-kr25"
server_key: "f86263137e8ff51eb11802a98d7647cb"
server_key_iv: "d5251bca63a71a9a7ec5959b9795269d"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.265842 18259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:47.269037 18273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:47.270265 18259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:47.270402 18259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "6e77322e29984269b2f2391dbd638fe8"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-kr25"
server_key: "f86263137e8ff51eb11802a98d7647cb"
server_key_iv: "d5251bca63a71a9a7ec5959b9795269d"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.270566 18259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:47.291061 18259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:47.291910 18259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:47.292150 18259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:47.293049 18259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:47.294618 18259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:47.294720 18259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:47.294803 18259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:47.294857 18259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:47.324321 18259 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41571
I20260927 22:51:47.324397 18385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41571 every 8 connection(s)
I20260927 22:51:47.325634 18259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:47.330492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18259
I20260927 22:51:47.330657 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:47.331064 388 external_mini_cluster.cc:1428] Setting key d248493954a5df349b322883a75c6de1
I20260927 22:51:47.334247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35445
--builtin_ntp_servers=127.0.97.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:51:47.338635 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.328149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54967 (local address 127.0.97.62:35445)
0927 22:51:47.328341 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.328345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.329825 (+ 1480us) server_negotiation.cc:407] Connection header received
0927 22:51:47.331725 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.331730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.331813 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.331942 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:47.333340 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.334350 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.335938 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.336488 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.336601 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.337900 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:47.337905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:47.337909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:47.337930 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:47.337968 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:47.337981 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:47.338061 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:47.338235 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 22:51:47.338332 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:47.339640 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.328645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35445 (local address 127.0.97.2:54967)
0927 22:51:47.329654 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:47.329697 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:47.331402 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:47.332204 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:47.332216 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:47.332359 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:47.333161 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.333188 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.334540 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.334544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.335806 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.335818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.336694 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.336699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.336912 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.337593 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:47.337596 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:47.337635 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:47.337717 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:47.338062 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:47.338068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:47.338073 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:51:47.338204 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 22:51:47.338501 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":299,"threads_started":1}
I20260927 22:51:47.341183 18386 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35445
I20260927 22:51:47.341502 18386 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:47.342378 18386 heartbeater.cc:511] Master 127.0.97.62:35445 requested a full tablet report, sending...
I20260927 22:51:47.343941 18069 ts_manager.cc:194] Registered new tserver with Master: 6e77322e29984269b2f2391dbd638fe8 (127.0.97.2:41571)
I20260927 22:51:47.345038 18069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:54967
W20260927 22:51:47.647533 18389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:47.647878 18389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:47.648017 18389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:47.655390 18389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:47.655521 18389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:47.655668 18389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:47.664662 18389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:47.665911 18389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:47.667693 18389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:47.680208 18394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:47.680167 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:47.680243 18395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:47.684851 18389 server_base.cc:1034] running on GCE node
I20260927 22:51:47.685439 18389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:47.686370 18389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:47.687523 18389 hybrid_clock.cc:648] HybridClock initialized: now 1790549507687470 us; error 46 us; skew 500 ppm
I20260927 22:51:47.687821 18389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:47.690129 18389 webserver.cc:460] Webserver started at http://127.0.97.3:33983/ using document root <none> and password file <none>
I20260927 22:51:47.690814 18389 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:47.690920 18389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:47.691234 18389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:47.694185 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "262c83dfdbed4cfcb63f50798ecedc1d"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-kr25"
server_key: "61a491bff4388f11b74f83322f307ad4"
server_key_iv: "828a96c1c7160c52ae1a41b8bd9b1c5c"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.694998 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "262c83dfdbed4cfcb63f50798ecedc1d"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-kr25"
server_key: "61a491bff4388f11b74f83322f307ad4"
server_key_iv: "828a96c1c7160c52ae1a41b8bd9b1c5c"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.699961 18389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:51:47.703190 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:47.704538 18389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:47.704706 18389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "262c83dfdbed4cfcb63f50798ecedc1d"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-kr25"
server_key: "61a491bff4388f11b74f83322f307ad4"
server_key_iv: "828a96c1c7160c52ae1a41b8bd9b1c5c"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.704974 18389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:47.731866 18389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:47.732733 18389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:47.732996 18389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:47.733959 18389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:47.735427 18389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:47.735538 18389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:47.735648 18389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:47.735720 18389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:47.765542 18389 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45925
I20260927 22:51:47.765597 18515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45925 every 8 connection(s)
I20260927 22:51:47.766810 18389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:47.770252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18389
I20260927 22:51:47.770453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:47.770998 388 external_mini_cluster.cc:1428] Setting key 4b8ebb95de12a53b9d65a918051a50fe
I20260927 22:51:47.779287 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.769480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43703 (local address 127.0.97.62:35445)
0927 22:51:47.769665 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.769669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.771143 (+ 1474us) server_negotiation.cc:407] Connection header received
0927 22:51:47.773178 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.773183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.773264 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.773367 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:47.774512 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.775253 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.776807 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.777281 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.777393 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.778617 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:47.778622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:47.778625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:47.778643 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:47.778675 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:47.778689 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:47.778773 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:47.778954 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 22:51:47.779032 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:47.780223 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.769951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35445 (local address 127.0.97.3:43703)
0927 22:51:47.770980 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:47.771020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:47.772894 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:47.773638 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:47.773650 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:47.773760 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:47.774340 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.774363 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.775372 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.775375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.776649 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.776661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.777459 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.777463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.777649 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.778277 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:47.778279 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:47.778321 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:47.778394 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:47.778769 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:47.778775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:47.778780 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:51:47.778940 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:51:47.779212 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":340,"threads_started":1}
I20260927 22:51:47.781491 18516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35445
I20260927 22:51:47.781747 18516 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:47.782433 18516 heartbeater.cc:511] Master 127.0.97.62:35445 requested a full tablet report, sending...
I20260927 22:51:47.783735 18069 ts_manager.cc:194] Registered new tserver with Master: 262c83dfdbed4cfcb63f50798ecedc1d (127.0.97.3:45925)
I20260927 22:51:47.784564 18069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43703
I20260927 22:51:47.786022 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:47.795374 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.789704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.0.97.62:35445)
0927 22:51:47.789866 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.789872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.790196 (+ 324us) server_negotiation.cc:407] Connection header received
0927 22:51:47.790341 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.790345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.790413 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.790493 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:47.790965 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.791665 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.793167 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.793594 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.793704 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.794725 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:47.794730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:47.794733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:47.794750 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:47.794780 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:47.794792 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:47.794869 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:47.795017 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:51:47.795102 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:51:47.798763 18069 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:47.805548 18069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:47.839980 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.817608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41571 (local address 127.0.0.1:44100)
0927 22:51:47.830083 (+ 12475us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:47.830100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:47.830200 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:47.831637 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:47.831644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:47.831673 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:47.832093 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.832110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.834456 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.834461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.837204 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.837219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.838748 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.838755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.839217 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.839260 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:51:47.839381 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 22:51:47.839572 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12340,"mutex_wait_us":49,"thread_start_us":11443,"threads_started":1}
I20260927 22:51:47.840464 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.817243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45925 (local address 127.0.0.1:43644)
0927 22:51:47.828596 (+ 11353us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:47.828645 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:47.828854 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:47.832027 (+ 3173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:47.832036 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:47.832062 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:47.832461 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.832471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.835898 (+ 3427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.835903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.837956 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.837969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.839716 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.839723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.839925 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.839960 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:51:47.840017 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 22:51:47.840153 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11056,"mutex_wait_us":3,"thread_start_us":12552,"threads_started":1}
I20260927 22:51:47.840926 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.817980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.0.97.2:41571)
0927 22:51:47.830954 (+ 12974us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.830962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.830991 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:51:47.831090 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.831097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.831380 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.832339 (+ 959us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:47.832590 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.834298 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.837401 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.838550 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.838682 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.838948 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:47.840424 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0927 22:51:47.840593 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12798,"thread_start_us":12819,"threads_started":1}
I20260927 22:51:47.841444 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.817428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.0.97.3:45925)
0927 22:51:47.831457 (+ 14029us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.831465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.831497 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:51:47.831592 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.831600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.831823 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.832643 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:47.832922 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.835718 (+ 2796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.838151 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.839523 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.840783 (+ 1260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.840973 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:47.841043 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:51:47.841186 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13852,"thread_start_us":13709,"threads_started":1}
I20260927 22:51:47.842649 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.831530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:58936)
0927 22:51:47.835130 (+ 3600us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:47.835143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:47.835269 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:47.835833 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:47.835837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:47.835863 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:47.836283 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.836292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.837665 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.837670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.839875 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.839888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.841878 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.841885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.842045 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.842076 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:51:47.842170 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:51:47.842296 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3484,"mutex_wait_us":488,"thread_start_us":3347,"threads_started":1}
I20260927 22:51:47.843230 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.830474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.0.97.1:46675)
0927 22:51:47.834748 (+ 4274us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.834757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.835251 (+ 494us) server_negotiation.cc:407] Connection header received
0927 22:51:47.835408 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.835413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.835645 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.835838 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:47.836458 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.837532 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.840386 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.841699 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.841832 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.842043 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:47.842223 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 22:51:47.843016 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4136,"thread_start_us":4042,"threads_started":1}
I20260927 22:51:47.844897 18451 tablet_service.cc:1467] Processing CreateTablet for tablet d3448b77ca2542f7b2ad8ca138bfb1ab (DEFAULT_TABLE table=test-table [id=f039bec0f09641abb7dd15c06f0f5cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:47.844897 18321 tablet_service.cc:1467] Processing CreateTablet for tablet d3448b77ca2542f7b2ad8ca138bfb1ab (DEFAULT_TABLE table=test-table [id=f039bec0f09641abb7dd15c06f0f5cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:47.846309 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3448b77ca2542f7b2ad8ca138bfb1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:47.846306 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3448b77ca2542f7b2ad8ca138bfb1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:47.846083 18190 tablet_service.cc:1467] Processing CreateTablet for tablet d3448b77ca2542f7b2ad8ca138bfb1ab (DEFAULT_TABLE table=test-table [id=f039bec0f09641abb7dd15c06f0f5cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:47.847201 18190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3448b77ca2542f7b2ad8ca138bfb1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:47.859325 18534 tablet_bootstrap.cc:492] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: Bootstrap starting.
I20260927 22:51:47.861094 18535 tablet_bootstrap.cc:492] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Bootstrap starting.
I20260927 22:51:47.862875 18534 tablet_bootstrap.cc:654] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:47.864112 18534 log.cc:824] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:47.864452 18535 tablet_bootstrap.cc:654] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:47.865532 18536 tablet_bootstrap.cc:492] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Bootstrap starting.
I20260927 22:51:47.865617 18535 log.cc:824] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:47.867341 18534 tablet_bootstrap.cc:492] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: No bootstrap required, opened a new log
I20260927 22:51:47.867726 18534 ts_tablet_manager.cc:1397] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 22:51:47.868798 18536 tablet_bootstrap.cc:654] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:47.869639 18535 tablet_bootstrap.cc:492] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: No bootstrap required, opened a new log
I20260927 22:51:47.869912 18535 ts_tablet_manager.cc:1397] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:51:47.869922 18536 log.cc:824] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:47.872725 18536 tablet_bootstrap.cc:492] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: No bootstrap required, opened a new log
I20260927 22:51:47.872967 18536 ts_tablet_manager.cc:1397] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:51:47.875281 18534 raft_consensus.cc:348] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:47.875655 18534 raft_consensus.cc:374] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:47.875747 18534 raft_consensus.cc:729] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf697025d847465ca768af4ae68d0c8a, State: Initialized, Role: FOLLOWER
I20260927 22:51:47.876291 18534 consensus_queue.cc:260] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:47.877270 18255 heartbeater.cc:503] Master 127.0.97.62:35445 was elected leader, sending a full tablet report...
I20260927 22:51:47.877923 18534 ts_tablet_manager.cc:1428] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 22:51:47.879637 18535 raft_consensus.cc:348] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:47.880046 18535 raft_consensus.cc:374] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:47.880137 18535 raft_consensus.cc:729] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262c83dfdbed4cfcb63f50798ecedc1d, State: Initialized, Role: FOLLOWER
I20260927 22:51:47.880700 18535 consensus_queue.cc:260] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:47.882974 18535 ts_tablet_manager.cc:1428] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 22:51:47.883162 18516 heartbeater.cc:503] Master 127.0.97.62:35445 was elected leader, sending a full tablet report...
I20260927 22:51:47.883781 18536 raft_consensus.cc:348] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:47.884089 18536 raft_consensus.cc:374] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:47.884171 18536 raft_consensus.cc:729] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e77322e29984269b2f2391dbd638fe8, State: Initialized, Role: FOLLOWER
I20260927 22:51:47.884698 18536 consensus_queue.cc:260] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:47.885715 18386 heartbeater.cc:503] Master 127.0.97.62:35445 was elected leader, sending a full tablet report...
I20260927 22:51:47.886262 18536 ts_tablet_manager.cc:1428] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260927 22:51:47.900303 18256 tablet.cc:2369] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:48.018493 18517 tablet.cc:2369] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:48.077560 18387 tablet.cc:2369] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:48.244311 18540 raft_consensus.cc:482] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:48.244660 18540 raft_consensus.cc:504] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:48.246289 18540 leader_election.cc:288] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 262c83dfdbed4cfcb63f50798ecedc1d (127.0.97.3:45925), 6e77322e29984269b2f2391dbd638fe8 (127.0.97.2:41571)
I20260927 22:51:48.249076 18542 raft_consensus.cc:482] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:48.249420 18542 raft_consensus.cc:504] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:48.251262 18542 leader_election.cc:288] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 262c83dfdbed4cfcb63f50798ecedc1d (127.0.97.3:45925), bf697025d847465ca768af4ae68d0c8a (127.0.97.1:46675)
I20260927 22:51:48.251816 18541 raft_consensus.cc:482] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:48.252104 18541 raft_consensus.cc:504] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:48.254045 18541 leader_election.cc:288] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf697025d847465ca768af4ae68d0c8a (127.0.97.1:46675), 6e77322e29984269b2f2391dbd638fe8 (127.0.97.2:41571)
I20260927 22:51:48.255200 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.246789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58031 (local address 127.0.97.2:41571)
0927 22:51:48.247115 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.247123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.247892 (+ 769us) server_negotiation.cc:407] Connection header received
0927 22:51:48.248062 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.248066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.248137 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.248213 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:48.248829 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.249688 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.252419 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.253648 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.253733 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.253833 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.254776 (+ 943us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.254922 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260927 22:51:48.255805 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.246720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51961 (local address 127.0.97.3:45925)
0927 22:51:48.246915 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.246922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.247948 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 22:51:48.248100 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.248104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.248166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.248239 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:48.248905 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.249955 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.252438 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.253632 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.253737 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.253833 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.255459 (+ 1626us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.255593 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:51:48.261471 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.246608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41571 (local address 127.0.97.1:58031)
0927 22:51:48.247752 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.247768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:48.247924 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:48.248245 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:48.248249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:48.248272 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:48.248656 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.248669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.249858 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.249863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.252193 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.252209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.254048 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.254055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.254236 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.254270 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:51:48.255491 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0927 22:51:48.255642 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":103,"thread_start_us":775,"threads_started":1}
I20260927 22:51:48.262876 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.254591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.97.3:44731)
0927 22:51:48.254860 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.254876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:48.254963 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:48.255993 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:48.255999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:48.256031 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:48.256230 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.256240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.257913 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.257919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.260136 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.260151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.261886 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.261893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.262128 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.262159 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:51:48.262238 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:51:48.262476 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260927 22:51:48.263041 18341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "bf697025d847465ca768af4ae68d0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e77322e29984269b2f2391dbd638fe8" is_pre_election: true
I20260927 22:51:48.263620 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.255390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44731 (local address 127.0.97.1:46675)
0927 22:51:48.255569 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.255577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.255594 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:51:48.255647 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.255651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.255753 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.255840 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:48.256474 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.257745 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.260330 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.261592 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.261697 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.261792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.263162 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.263310 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":68}
I20260927 22:51:48.263571 18341 raft_consensus.cc:2454] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf697025d847465ca768af4ae68d0c8a in term 0.
I20260927 22:51:48.264460 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.251821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60371 (local address 127.0.97.3:45925)
0927 22:51:48.255999 (+ 4178us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.256004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.256019 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:51:48.256099 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.256103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.256169 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.256247 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:48.257311 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.258597 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.261218 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.262642 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.262740 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.262832 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.264072 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.264191 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4076,"threads_started":1}
I20260927 22:51:48.264535 18146 leader_election.cc:302] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e77322e29984269b2f2391dbd638fe8, bf697025d847465ca768af4ae68d0c8a; no voters:
I20260927 22:51:48.265002 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.251551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45925 (local address 127.0.97.2:60371)
0927 22:51:48.254460 (+ 2909us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.254504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:48.254625 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:48.256749 (+ 2124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:48.256754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:48.256785 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:48.257148 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.257159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.258799 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.258805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.261001 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.261017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.263016 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.263023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.263224 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.263255 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:51:48.264584 (+ 1329us) client_negotiation.cc:240] Negotiation successful
0927 22:51:48.264730 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2740,"thread_start_us":2580,"threads_started":1}
I20260927 22:51:48.265027 18540 raft_consensus.cc:2781] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:48.265178 18540 raft_consensus.cc:482] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:48.265288 18540 raft_consensus.cc:3037] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:48.268850 18540 raft_consensus.cc:504] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:48.270260 18341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "bf697025d847465ca768af4ae68d0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e77322e29984269b2f2391dbd638fe8"
I20260927 22:51:48.270471 18341 raft_consensus.cc:3037] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:48.271130 18540 leader_election.cc:288] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [CANDIDATE]: Term 1 election: Requested vote from peers 262c83dfdbed4cfcb63f50798ecedc1d (127.0.97.3:45925), 6e77322e29984269b2f2391dbd638fe8 (127.0.97.2:41571)
I20260927 22:51:48.271359 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "6e77322e29984269b2f2391dbd638fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" is_pre_election: true
I20260927 22:51:48.271703 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.246733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45925 (local address 127.0.97.1:51961)
0927 22:51:48.247808 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.247825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:48.247961 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:48.248270 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:48.248274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:48.248296 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:48.248735 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.248747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.250090 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.250099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.252231 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.252241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.254412 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.254420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.254601 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.254632 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:51:48.256492 (+ 1860us) client_negotiation.cc:240] Negotiation successful
0927 22:51:48.268686 (+ 12194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":11864,"thread_start_us":544,"threads_started":1}
I20260927 22:51:48.271814 18471 raft_consensus.cc:2454] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e77322e29984269b2f2391dbd638fe8 in term 0.
I20260927 22:51:48.272308 18470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "bf697025d847465ca768af4ae68d0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" is_pre_election: true
I20260927 22:51:48.272575 18470 raft_consensus.cc:2454] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf697025d847465ca768af4ae68d0c8a in term 0.
I20260927 22:51:48.272501 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf697025d847465ca768af4ae68d0c8a" is_pre_election: true
I20260927 22:51:48.272935 18210 raft_consensus.cc:2379] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 262c83dfdbed4cfcb63f50798ecedc1d in current term 1: Already voted for candidate bf697025d847465ca768af4ae68d0c8a in this term.
I20260927 22:51:48.273216 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.255528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47725 (local address 127.0.97.1:46675)
0927 22:51:48.263853 (+ 8325us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.263859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.263876 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:51:48.263932 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.263937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.264021 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.264110 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:48.264769 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.265958 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.269084 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.270359 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.270457 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.270553 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.272827 (+ 2274us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.272953 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8198,"thread_start_us":15039,"threads_started":1}
I20260927 22:51:48.273360 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.254563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.97.2:47725)
0927 22:51:48.255215 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.255234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:48.255355 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:48.264129 (+ 8774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:48.264134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:48.264159 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:48.264585 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.264599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.266151 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.266157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.268305 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.268320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.270791 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.270800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.271776 (+ 976us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.271808 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:51:48.272366 (+ 558us) client_negotiation.cc:240] Negotiation successful
0927 22:51:48.272511 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1,"thread_start_us":333,"threads_started":1}
I20260927 22:51:48.273383 18275 leader_election.cc:302] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 262c83dfdbed4cfcb63f50798ecedc1d, 6e77322e29984269b2f2391dbd638fe8; no voters:
I20260927 22:51:48.273737 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "bf697025d847465ca768af4ae68d0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "262c83dfdbed4cfcb63f50798ecedc1d"
I20260927 22:51:48.273909 18471 raft_consensus.cc:3037] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:48.274673 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.256508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36667 (local address 127.0.97.2:41571)
0927 22:51:48.256640 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.256646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.263847 (+ 7201us) server_negotiation.cc:407] Connection header received
0927 22:51:48.265217 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.265222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.265315 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.265380 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:48.266084 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.267174 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.270161 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.271419 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.271555 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.271656 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.274343 (+ 2687us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.274468 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":30}
I20260927 22:51:48.274923 18341 raft_consensus.cc:2454] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf697025d847465ca768af4ae68d0c8a in term 1.
I20260927 22:51:48.275121 18542 raft_consensus.cc:2741] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 22:51:48.275213 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.256157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41571 (local address 127.0.97.3:36667)
0927 22:51:48.263756 (+ 7599us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.263779 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:48.263941 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:48.265555 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:48.265559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:48.265587 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:48.265902 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.265915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.267525 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.267530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.269749 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:48.269762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.274019 (+ 4257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.274024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:48.274173 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.274201 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:51:48.274812 (+ 611us) client_negotiation.cc:240] Negotiation successful
0927 22:51:48.274957 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7475,"thread_start_us":15124,"threads_started":1}
I20260927 22:51:48.275313 18146 leader_election.cc:302] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e77322e29984269b2f2391dbd638fe8, bf697025d847465ca768af4ae68d0c8a; no voters:
I20260927 22:51:48.276013 18540 raft_consensus.cc:2781] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:48.276259 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "6e77322e29984269b2f2391dbd638fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf697025d847465ca768af4ae68d0c8a" is_pre_election: true
I20260927 22:51:48.276160 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3448b77ca2542f7b2ad8ca138bfb1ab" candidate_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e77322e29984269b2f2391dbd638fe8" is_pre_election: true
I20260927 22:51:48.276549 18340 raft_consensus.cc:2379] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 262c83dfdbed4cfcb63f50798ecedc1d in current term 1: Already voted for candidate bf697025d847465ca768af4ae68d0c8a in this term.
I20260927 22:51:48.277132 18471 raft_consensus.cc:2454] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf697025d847465ca768af4ae68d0c8a in term 1.
I20260927 22:51:48.277132 18540 raft_consensus.cc:686] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 1 LEADER]: Becoming Leader. State: Replica: bf697025d847465ca768af4ae68d0c8a, State: Running, Role: LEADER
I20260927 22:51:48.277213 18407 leader_election.cc:302] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 262c83dfdbed4cfcb63f50798ecedc1d; no voters: 6e77322e29984269b2f2391dbd638fe8, bf697025d847465ca768af4ae68d0c8a
I20260927 22:51:48.277742 18541 raft_consensus.cc:2726] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:51:48.277886 18540 consensus_queue.cc:237] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } }
I20260927 22:51:48.282580 18068 catalog_manager.cc:5495] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to bf697025d847465ca768af4ae68d0c8a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bf697025d847465ca768af4ae68d0c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf697025d847465ca768af4ae68d0c8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:48.403273 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.397982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.0.97.1:46675)
0927 22:51:48.398175 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.398180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.398528 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:51:48.398704 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.398708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.398790 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.398917 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:48.399270 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.400009 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.401345 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.401771 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.401865 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.402311 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:48.402825 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:48.402941 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.403018 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.403081 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:51:48.410889 18340 raft_consensus.cc:1261] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 1 FOLLOWER]: Refusing update from remote peer bf697025d847465ca768af4ae68d0c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:48.410967 18471 raft_consensus.cc:1261] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 1 FOLLOWER]: Refusing update from remote peer bf697025d847465ca768af4ae68d0c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:48.411831 18550 consensus_queue.cc:1035] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e77322e29984269b2f2391dbd638fe8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:48.412225 18540 consensus_queue.cc:1035] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "262c83dfdbed4cfcb63f50798ecedc1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:48.427541 18556 mvcc.cc:205] Tried to move back new op lower bound from 7334090786441019392 to 7334090785909510144. Current Snapshot: MvccSnapshot[applied={T|T < 7334090786441019392}]
I20260927 22:51:48.435848 18068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:48.436131 18068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:48.439491 18068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72caea45cab444c49acf74787edec685: Sending DeleteTablet for 3 replicas of tablet d3448b77ca2542f7b2ad8ca138bfb1ab
I20260927 22:51:48.440508 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet d3448b77ca2542f7b2ad8ca138bfb1ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:48 UTC) from {username='slave'} at 127.0.0.1:43644
I20260927 22:51:48.440569 18190 tablet_service.cc:1514] Processing DeleteTablet for tablet d3448b77ca2542f7b2ad8ca138bfb1ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:48 UTC) from {username='slave'} at 127.0.0.1:58936
I20260927 22:51:48.440662 18321 tablet_service.cc:1514] Processing DeleteTablet for tablet d3448b77ca2542f7b2ad8ca138bfb1ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:48 UTC) from {username='slave'} at 127.0.0.1:44100
I20260927 22:51:48.448514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18128
I20260927 22:51:48.453240 18564 tablet_replica.cc:330] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: stopping tablet replica
I20260927 22:51:48.453948 18564 raft_consensus.cc:2229] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:48.454710 18564 raft_consensus.cc:2258] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:48.457028 18564 ts_tablet_manager.cc:1905] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:48.460588 18566 tablet_replica.cc:330] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: stopping tablet replica
I20260927 22:51:48.461061 18566 raft_consensus.cc:2229] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:48.461591 18566 raft_consensus.cc:2258] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:48.464272 18564 ts_tablet_manager.cc:1918] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:48.464437 18564 log.cc:1196] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/d3448b77ca2542f7b2ad8ca138bfb1ab
I20260927 22:51:48.464927 18564 ts_tablet_manager.cc:1939] T d3448b77ca2542f7b2ad8ca138bfb1ab P 262c83dfdbed4cfcb63f50798ecedc1d: Deleting consensus metadata
I20260927 22:51:48.465233 18566 ts_tablet_manager.cc:1905] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:48.466727 18054 catalog_manager.cc:4841] TS 262c83dfdbed4cfcb63f50798ecedc1d (127.0.97.3:45925): tablet d3448b77ca2542f7b2ad8ca138bfb1ab (table test-table [id=f039bec0f09641abb7dd15c06f0f5cb1]) successfully deleted
I20260927 22:51:48.469415 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.451712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.0.97.1:46675)
0927 22:51:48.451948 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.451956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.455891 (+ 3935us) server_negotiation.cc:407] Connection header received
0927 22:51:48.456232 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.456238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.456346 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.456422 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:48.457098 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.458153 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.462041 (+ 3888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.462796 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.462903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.467598 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:48.467611 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:48.467630 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:48.468411 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:48.468502 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:48.468528 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:48.468616 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.468875 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.469010 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:48.470484 18565 tablet_replica.cc:330] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a: stopping tablet replica
I20260927 22:51:48.470641 18250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:48.471720 18566 ts_tablet_manager.cc:1918] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:48.471932 18566 log.cc:1196] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/d3448b77ca2542f7b2ad8ca138bfb1ab
I20260927 22:51:48.472402 18566 ts_tablet_manager.cc:1939] T d3448b77ca2542f7b2ad8ca138bfb1ab P 6e77322e29984269b2f2391dbd638fe8: Deleting consensus metadata
I20260927 22:51:48.474141 18056 catalog_manager.cc:4841] TS 6e77322e29984269b2f2391dbd638fe8 (127.0.97.2:41571): tablet d3448b77ca2542f7b2ad8ca138bfb1ab (table test-table [id=f039bec0f09641abb7dd15c06f0f5cb1]) successfully deleted
I20260927 22:51:48.470934 18565 raft_consensus.cc:2229] T d3448b77ca2542f7b2ad8ca138bfb1ab P bf697025d847465ca768af4ae68d0c8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:48.566109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18128
I20260927 22:51:48.580968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18259
W20260927 22:51:48.581384 18054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46675 (error 108)
I20260927 22:51:48.583343 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.582640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:58962)
0927 22:51:48.583080 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.583176 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":207,"threads_started":1}
W20260927 22:51:48.583580 18054 catalog_manager.cc:4584] TS bf697025d847465ca768af4ae68d0c8a (127.0.97.1:46675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3448b77ca2542f7b2ad8ca138bfb1ab: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
I20260927 22:51:48.587666 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.581781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.0.97.2:41571)
0927 22:51:48.581939 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.581944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.581984 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:51:48.582202 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.582208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.582301 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.582401 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:48.582799 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.583593 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.584882 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.585261 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.585366 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.586324 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:48.586335 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:48.586350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:48.586889 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:48.586936 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:48.586961 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:48.587053 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.587184 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.587287 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:51:48.588241 18381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:48.637432 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.636863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:58976)
0927 22:51:48.637132 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.637259 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 22:51:48.638528 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.638112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:58984)
0927 22:51:48.638259 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.638361 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 22:51:48.679487 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18259
I20260927 22:51:48.693816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18389
I20260927 22:51:48.698879 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.698438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:58998)
0927 22:51:48.698624 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.698708 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:51:48.700070 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.699647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:59000)
0927 22:51:48.699827 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.699910 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:51:48.704607 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.698292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.0.97.3:45925)
0927 22:51:48.698480 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:48.698485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:48.698556 (+ 71us) server_negotiation.cc:407] Connection header received
0927 22:51:48.698743 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:48.698748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:48.698856 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:48.698993 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:48.699604 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.700591 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.701891 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:48.702284 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:48.702371 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:48.703373 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:48.703381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:48.703396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:48.703998 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:48.704043 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:48.704061 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:48.704147 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:48.704242 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:51:48.704315 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:51:48.705257 18511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:48.780809 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.780318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:59006)
0927 22:51:48.780549 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.780641 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:51:48.781903 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:48.781485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46675 (local address 127.0.0.1:59012)
0927 22:51:48.781664 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:48.781733 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 22:51:48.793027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18389
I20260927 22:51:48.806143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18038
I20260927 22:51:48.807173 18099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:48.912325 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18038
2026-09-27T22:51:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3046 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:49 dist-test-slave-kr25 krb5kdc[18576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:49 dist-test-slave-kr25 krb5kdc[18576](info): set up 2 sockets
Sep 27 22:51:49 dist-test-slave-kr25 krb5kdc[18576](info): commencing operation
krb5kdc: starting...
W20260927 22:51:49.919135 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:50 dist-test-slave-kr25 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:50.146552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40973
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:50.464315 18589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:50.464605 18589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:50.464735 18589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:50.472048 18589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:50.472157 18589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:50.472230 18589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:50.472306 18589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:50.472368 18589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:50.482474 18589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:50.483799 18589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:50.485093 18589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:50.496248 18595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:50.496337 18597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:50.496376 18594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:50.497479 18589 server_base.cc:1034] running on GCE node
I20260927 22:51:50.498323 18589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:50.499775 18589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:50.501030 18589 hybrid_clock.cc:648] HybridClock initialized: now 1790549510500976 us; error 36 us; skew 500 ppm
I20260927 22:51:50.501398 18589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:50 dist-test-slave-kr25 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:50.504464 18589 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:50.505947 18589 webserver.cc:460] Webserver started at http://127.0.97.62:43095/ using document root <none> and password file <none>
I20260927 22:51:50.506609 18589 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:50.506721 18589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:50.507007 18589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:50.509919 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "aaed8c4c553e4e90b26cf04e09162166"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-kr25"
server_key: "04d82fb4b53ef5690ab5f89b2e27dd67"
server_key_iv: "f6b4a9d0815dc50534e955aa5cf4d4ea"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.510609 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "aaed8c4c553e4e90b26cf04e09162166"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-kr25"
server_key: "04d82fb4b53ef5690ab5f89b2e27dd67"
server_key_iv: "f6b4a9d0815dc50534e955aa5cf4d4ea"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.515407 18589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:51:50.518555 18604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:50.519781 18589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:50.519953 18589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "aaed8c4c553e4e90b26cf04e09162166"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-kr25"
server_key: "04d82fb4b53ef5690ab5f89b2e27dd67"
server_key_iv: "f6b4a9d0815dc50534e955aa5cf4d4ea"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.520115 18589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:50.536540 18589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:50.540498 18589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:50.540759 18589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:50.557556 18589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40973
I20260927 22:51:50.557591 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40973 every 8 connection(s)
I20260927 22:51:50.558851 18589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:50.562278 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:50.568439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18589
I20260927 22:51:50.568604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:50.569016 388 external_mini_cluster.cc:1428] Setting key 2ef2059e9f14df43209fd2b1040df74d
I20260927 22:51:50.572676 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166: Bootstrap starting.
I20260927 22:51:50.575747 18656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:50.576764 18656 log.cc:824] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:50.579780 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166: No bootstrap required, opened a new log
Sep 27 22:51:50 dist-test-slave-kr25 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:50.587493 18656 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaed8c4c553e4e90b26cf04e09162166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40973 } }
I20260927 22:51:50.587960 18656 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:50.588068 18656 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaed8c4c553e4e90b26cf04e09162166, State: Initialized, Role: FOLLOWER
I20260927 22:51:50.588609 18656 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaed8c4c553e4e90b26cf04e09162166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40973 } }
I20260927 22:51:50.588822 18656 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:50.588958 18656 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:50.589118 18656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:50.591176 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:50.570887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.0.97.62:40973)
0927 22:51:50.571766 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:50.571779 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:50.571828 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:51:50.573616 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:50.573640 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:50.574051 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:50.574416 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:50.575905 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.577305 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.578995 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.579997 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.580169 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:50.584160 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:50.584168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:50.584185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:50.584224 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:50.587080 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:50.587903 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:50.587933 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:50.587940 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:50.588131 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:50.588514 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:50.588519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:50.588523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:50.589128 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:50.589389 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:50.589801 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 22:51:50.590194 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":347,"threads_started":1}
I20260927 22:51:50.591691 18656 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaed8c4c553e4e90b26cf04e09162166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40973 } }
I20260927 22:51:50.592161 18656 leader_election.cc:302] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaed8c4c553e4e90b26cf04e09162166; no voters:
I20260927 22:51:50.592752 18656 leader_election.cc:288] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:50.592865 18661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:50.593808 18661 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [term 1 LEADER]: Becoming Leader. State: Replica: aaed8c4c553e4e90b26cf04e09162166, State: Running, Role: LEADER
I20260927 22:51:50.594395 18661 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaed8c4c553e4e90b26cf04e09162166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40973 } }
I20260927 22:51:50.594837 18656 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:50.597158 18662 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaed8c4c553e4e90b26cf04e09162166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaed8c4c553e4e90b26cf04e09162166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40973 } } }
I20260927 22:51:50.597543 18662 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:50.596894 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaed8c4c553e4e90b26cf04e09162166. Latest consensus state: current_term: 1 leader_uuid: "aaed8c4c553e4e90b26cf04e09162166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaed8c4c553e4e90b26cf04e09162166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40973 } } }
I20260927 22:51:50.598176 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:50.604174 18670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:50.608899 18670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:50.618067 18670 catalog_manager.cc:1347] Generated new cluster ID: f2b28f33ddcd49b08f2df3db6008fabc
I20260927 22:51:50.618177 18670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:50.627962 18670 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:50.629372 18670 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:50.636878 18670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166: Generated new TSK 0
I20260927 22:51:50.637598 18670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:50.691244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40973
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:51.005918 18683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:51.006217 18683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:51.006323 18683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:51.013512 18683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:51.013634 18683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:51.013757 18683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:51.023603 18683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:51.024863 18683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:51.026082 18683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:51.038354 18688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.038393 18689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.038826 18691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:51.039284 18683 server_base.cc:1034] running on GCE node
I20260927 22:51:51.039902 18683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:51.040777 18683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:51.041986 18683 hybrid_clock.cc:648] HybridClock initialized: now 1790549511041943 us; error 37 us; skew 500 ppm
I20260927 22:51:51.042263 18683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:51 dist-test-slave-kr25 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:51.045179 18683 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:51.046577 18683 webserver.cc:460] Webserver started at http://127.0.97.1:41649/ using document root <none> and password file <none>
I20260927 22:51:51.047224 18683 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:51.047322 18683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:51.047632 18683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:51.050542 18683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "cf6e19c1c27e487abab46986729ac551"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-kr25"
server_key: "0a1c25b55d061f0d9ad5c3157c6da31f"
server_key_iv: "13cde7e07e8767f4efe1817388f985e0"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.051373 18683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cf6e19c1c27e487abab46986729ac551"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-kr25"
server_key: "0a1c25b55d061f0d9ad5c3157c6da31f"
server_key_iv: "13cde7e07e8767f4efe1817388f985e0"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.056797 18683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:51:51.060271 18698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.061587 18683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:51:51.061734 18683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "cf6e19c1c27e487abab46986729ac551"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-kr25"
server_key: "0a1c25b55d061f0d9ad5c3157c6da31f"
server_key_iv: "13cde7e07e8767f4efe1817388f985e0"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.061924 18683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:51.073365 18683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:51.076915 18683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:51.077157 18683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:51.078018 18683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:51.079249 18683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:51.079321 18683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.079413 18683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:51.079494 18683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 22:51:51.108891 18683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44045
I20260927 22:51:51.108932 18810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44045 every 8 connection(s)
I20260927 22:51:51.110098 18683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:51.113412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18683
I20260927 22:51:51.113598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:51.114133 388 external_mini_cluster.cc:1428] Setting key 20360f9f772c3527b0ffe93f56478935
Sep 27 22:51:51 dist-test-slave-kr25 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:51.130213 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.112473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41403 (local address 127.0.97.62:40973)
0927 22:51:51.112893 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:51.112898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:51.113940 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 22:51:51.116127 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:51.116133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:51.116210 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:51.116317 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:51.117748 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.118751 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.120247 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.120798 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.120911 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.124387 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:51.124391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:51.124395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:51.124457 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:51.127282 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.127999 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.128004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.128007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.128141 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.128507 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.128511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.128514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.128848 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:51.128992 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:51.129630 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 22:51:51.129869 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":212,"threads_started":1}
I20260927 22:51:51.131071 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.113007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40973 (local address 127.0.97.1:41403)
0927 22:51:51.113789 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:51.113828 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:51.115837 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:51.116536 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:51.116544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:51.117027 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:51.117579 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.117597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.118869 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.118873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.120121 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.120137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.120980 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.120984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.121172 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.121798 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:51.121819 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:51.124178 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:51.127414 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.127422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.127462 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.127873 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.128229 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.128232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.128234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.128407 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.128967 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:51.128972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:51.129283 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 22:51:51.129506 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 22:51:51.129921 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":301,"threads_started":1}
I20260927 22:51:51.132704 18811 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40973
I20260927 22:51:51.133019 18811 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:51.133709 18811 heartbeater.cc:511] Master 127.0.97.62:40973 requested a full tablet report, sending...
I20260927 22:51:51.135831 18621 ts_manager.cc:194] Registered new tserver with Master: cf6e19c1c27e487abab46986729ac551 (127.0.97.1:44045)
I20260927 22:51:51.137481 18621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41403
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:51.166501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40973
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:51.483158 18817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:51.483453 18817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:51.483548 18817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:51.490726 18817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:51.490864 18817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:51.490981 18817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:51.501111 18817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:51.502355 18817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:51.503645 18817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:51.515864 18823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.515836 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.516116 18825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:51.516551 18817 server_base.cc:1034] running on GCE node
I20260927 22:51:51.517105 18817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:51.518033 18817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:51.519230 18817 hybrid_clock.cc:648] HybridClock initialized: now 1790549511519161 us; error 60 us; skew 500 ppm
I20260927 22:51:51.519563 18817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:51 dist-test-slave-kr25 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:51.522459 18817 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:51.523847 18817 webserver.cc:460] Webserver started at http://127.0.97.2:33593/ using document root <none> and password file <none>
I20260927 22:51:51.524533 18817 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:51.524645 18817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:51.524923 18817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:51.527699 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "658d69fbd6c9455c966acea5a2e5d745"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-kr25"
server_key: "4e081c9aae5ad3f473b1a76a901a2794"
server_key_iv: "2bbf5fa7b3b554fb3677afc27ea3e6be"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.528676 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "658d69fbd6c9455c966acea5a2e5d745"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-kr25"
server_key: "4e081c9aae5ad3f473b1a76a901a2794"
server_key_iv: "2bbf5fa7b3b554fb3677afc27ea3e6be"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.533869 18817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:51:51.537468 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.538862 18817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:51.539021 18817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "658d69fbd6c9455c966acea5a2e5d745"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-kr25"
server_key: "4e081c9aae5ad3f473b1a76a901a2794"
server_key_iv: "2bbf5fa7b3b554fb3677afc27ea3e6be"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.539201 18817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:51.550280 18817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:51.554142 18817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:51.554389 18817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:51.555260 18817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:51.556504 18817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:51.556578 18817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.556691 18817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:51.556742 18817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.592228 18817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43275
I20260927 22:51:51.592274 18944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43275 every 8 connection(s)
I20260927 22:51:51.593430 18817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:51.599565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18817
I20260927 22:51:51.599740 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:51.600165 388 external_mini_cluster.cc:1428] Setting key 642236b08470f9de599b8d40ba300dbe
Sep 27 22:51:51 dist-test-slave-kr25 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:51.613353 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.595925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33689 (local address 127.0.97.62:40973)
0927 22:51:51.596129 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:51.596136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:51.597282 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 22:51:51.599079 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:51.599083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:51.599182 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:51.599281 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:51.600901 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.601686 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.603288 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.603872 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.604011 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.607214 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:51.607219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:51.607223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:51.607258 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:51.610066 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.611003 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.611008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.611011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.611165 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.611624 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.611628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.611631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.611980 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:51.612119 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:51.612804 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 22:51:51.613037 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:51:51.614275 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.596444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40973 (local address 127.0.97.2:33689)
0927 22:51:51.597129 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:51.597172 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:51.598850 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:51.599565 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:51.599573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:51.600101 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:51.600724 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.600749 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.601840 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.601846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.603134 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.603150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.604074 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.604079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.604265 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.604936 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:51.604957 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:51.607001 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:51.610217 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.610227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.610246 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.610881 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.611263 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.611266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.611268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.611509 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.612093 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:51.612099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:51.612450 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 22:51:51.612686 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 22:51:51.613091 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":263,"threads_started":1}
I20260927 22:51:51.615729 18945 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40973
I20260927 22:51:51.616027 18945 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:51.616722 18945 heartbeater.cc:511] Master 127.0.97.62:40973 requested a full tablet report, sending...
I20260927 22:51:51.618299 18621 ts_manager.cc:194] Registered new tserver with Master: 658d69fbd6c9455c966acea5a2e5d745 (127.0.97.2:43275)
I20260927 22:51:51.619172 18621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33689
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:51.652057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40973
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:51.963209 18950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:51.963485 18950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:51.963604 18950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:51.970630 18950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:51.970742 18950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:51.970907 18950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:51.980720 18950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:51.981971 18950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:51.983227 18950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:51.995517 18958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.995609 18955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.995582 18956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:51.996702 18950 server_base.cc:1034] running on GCE node
I20260927 22:51:51.997265 18950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:51.998191 18950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:51.999372 18950 hybrid_clock.cc:648] HybridClock initialized: now 1790549511999312 us; error 55 us; skew 500 ppm
I20260927 22:51:51.999697 18950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:52 dist-test-slave-kr25 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:52.009850 18950 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:52.011502 18950 webserver.cc:460] Webserver started at http://127.0.97.3:40095/ using document root <none> and password file <none>
I20260927 22:51:52.012228 18950 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:52.012337 18950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:52.012655 18950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:52.015619 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b"
format_stamp: "Formatted at 2026-09-27 22:51:52 on dist-test-slave-kr25"
server_key: "f784d09f786e5566692e81ea4ec9c1ff"
server_key_iv: "1bf12c24cd16765fc8a3438603fda942"
server_key_version: "encryptionkey@0"
I20260927 22:51:52.016372 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b"
format_stamp: "Formatted at 2026-09-27 22:51:52 on dist-test-slave-kr25"
server_key: "f784d09f786e5566692e81ea4ec9c1ff"
server_key_iv: "1bf12c24cd16765fc8a3438603fda942"
server_key_version: "encryptionkey@0"
I20260927 22:51:52.021528 18950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:51:52.024924 18965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:52.026228 18950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:52.026364 18950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b"
format_stamp: "Formatted at 2026-09-27 22:51:52 on dist-test-slave-kr25"
server_key: "f784d09f786e5566692e81ea4ec9c1ff"
server_key_iv: "1bf12c24cd16765fc8a3438603fda942"
server_key_version: "encryptionkey@0"
I20260927 22:51:52.026536 18950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:52.038699 18950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:52.042352 18950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:52.042603 18950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:52.043519 18950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:52.044767 18950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:52.044837 18950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:52.044950 18950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:52.045001 18950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:52.074997 18950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41743
I20260927 22:51:52.075049 19077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41743 every 8 connection(s)
I20260927 22:51:52.076215 18950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:52.083146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 18950
I20260927 22:51:52.083329 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:52.083861 388 external_mini_cluster.cc:1428] Setting key ddaefab552447f4c4304abc064e3ebd5
Sep 27 22:51:52 dist-test-slave-kr25 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:52.094908 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.078609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46511 (local address 127.0.97.62:40973)
0927 22:51:52.078809 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.078815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.079952 (+ 1137us) server_negotiation.cc:407] Connection header received
0927 22:51:52.081646 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.081650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.081715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.081800 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.083212 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.084278 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.085980 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.086513 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.086620 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.089740 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.089744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.089747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.089781 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.092177 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.092884 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.092887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.092890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.093006 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.093344 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.093347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.093349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.093628 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.093737 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.094409 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.094641 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:51:52.095906 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.079077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40973 (local address 127.0.97.3:46511)
0927 22:51:52.079764 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:52.079806 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:52.081406 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:52.082003 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:52.082011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:52.082472 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:52.083033 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.083051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.084441 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.084447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.085843 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.085859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.086702 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.086707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.086901 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.087536 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:52.087558 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:52.089554 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:52.092318 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:52.092324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:52.092338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:52.092790 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:52.093087 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:52.093089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:52.093092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:52.093267 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:52.093729 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:52.093735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:52.094064 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 22:51:52.094295 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 22:51:52.094725 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
I20260927 22:51:52.097405 19078 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40973
I20260927 22:51:52.097711 19078 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:52.098454 19078 heartbeater.cc:511] Master 127.0.97.62:40973 requested a full tablet report, sending...
I20260927 22:51:52.099768 18621 ts_manager.cc:194] Registered new tserver with Master: e8e66a354b4f403aa1a91ef8c0d93a7b (127.0.97.3:41743)
I20260927 22:51:52.100543 18621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46511
I20260927 22:51:52.110687 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:52.126277 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.114736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.0.97.62:40973)
0927 22:51:52.114936 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.114943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.115303 (+ 360us) server_negotiation.cc:407] Connection header received
0927 22:51:52.115423 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.115426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.115510 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.115586 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.116820 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.117505 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.118736 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.119126 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.119208 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.121538 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.121543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.121546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.121576 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.123966 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.124610 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.124615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.124618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.124737 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.125085 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.125088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.125090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.125364 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.125547 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.125847 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.125982 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:51:52.129768 18621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:52.136579 18621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:52.140295 18811 heartbeater.cc:503] Master 127.0.97.62:40973 was elected leader, sending a full tablet report...
I20260927 22:51:52.159089 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.148164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44045 (local address 127.0.0.1:33496)
0927 22:51:52.150177 (+ 2013us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:52.150211 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:52.150372 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:52.151484 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:52.151492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:52.151520 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:52.151976 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.151995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.154097 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.154102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.156312 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.156326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.158177 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.158182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.158369 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.158427 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:51:52.158525 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:51:52.158718 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794,"thread_start_us":2470,"threads_started":1}
I20260927 22:51:52.160661 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.148369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.0.97.1:44045)
0927 22:51:52.150849 (+ 2480us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.150858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.150886 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:51:52.150995 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.151002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.151248 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.152172 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:52.152396 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.153949 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.156523 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.157967 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.159853 (+ 1886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.160113 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.160185 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.160348 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2311,"thread_start_us":2160,"threads_started":1}
I20260927 22:51:52.162221 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.149066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.0.97.2:43275)
0927 22:51:52.149666 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.149676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.150995 (+ 1319us) server_negotiation.cc:407] Connection header received
0927 22:51:52.151243 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.151249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.151526 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.151683 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:52.153595 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.155164 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.158523 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.159956 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.160054 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.160212 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.161737 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.161925 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":355,"threads_started":1}
I20260927 22:51:52.163066 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.148592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43275 (local address 127.0.0.1:48834)
0927 22:51:52.150896 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:52.150916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:52.151088 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:52.152995 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:52.152999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:52.153023 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:52.153401 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.153424 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.155336 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.155341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.158345 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.158358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.161339 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.161346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.161524 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.161559 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:51:52.162466 (+ 907us) client_negotiation.cc:240] Negotiation successful
0927 22:51:52.162614 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2183,"mutex_wait_us":1,"thread_start_us":1238,"threads_started":1}
I20260927 22:51:52.166810 18746 tablet_service.cc:1467] Processing CreateTablet for tablet 16e6336a728c45869a18aace0ef5585a (DEFAULT_TABLE table=test-table [id=92afac30add64ea9bcd96583137bb40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:52.166836 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 16e6336a728c45869a18aace0ef5585a (DEFAULT_TABLE table=test-table [id=92afac30add64ea9bcd96583137bb40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:52.168123 18746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e6336a728c45869a18aace0ef5585a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:52.168304 18880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e6336a728c45869a18aace0ef5585a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:52.168787 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.150987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41743 (local address 127.0.0.1:38502)
0927 22:51:52.159326 (+ 8339us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:52.159347 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:52.159462 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:52.162463 (+ 3001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:52.162468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:52.162494 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:52.162652 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.162662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.164508 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.164513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.166292 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.166316 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.168043 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.168048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.168225 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.168280 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 22:51:52.168376 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:51:52.168525 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8246,"thread_start_us":11870,"threads_started":1}
I20260927 22:51:52.168934 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.151350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.0.97.3:41743)
0927 22:51:52.161820 (+ 10470us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.161828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.161860 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:51:52.161961 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.161969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.162230 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.162836 (+ 606us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:52.163110 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.164337 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.166491 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.167876 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.168006 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.168219 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.168444 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.168601 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10295,"thread_start_us":10117,"threads_started":1}
I20260927 22:51:52.172916 19013 tablet_service.cc:1467] Processing CreateTablet for tablet 16e6336a728c45869a18aace0ef5585a (DEFAULT_TABLE table=test-table [id=92afac30add64ea9bcd96583137bb40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:52.174556 19013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e6336a728c45869a18aace0ef5585a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:52.181052 19096 tablet_bootstrap.cc:492] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Bootstrap starting.
I20260927 22:51:52.183063 19097 tablet_bootstrap.cc:492] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Bootstrap starting.
I20260927 22:51:52.184777 19096 tablet_bootstrap.cc:654] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:52.186033 19096 log.cc:824] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:52.186419 19097 tablet_bootstrap.cc:654] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:52.187455 19097 log.cc:824] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:52.188130 19099 tablet_bootstrap.cc:492] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Bootstrap starting.
I20260927 22:51:52.190358 19096 tablet_bootstrap.cc:492] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: No bootstrap required, opened a new log
I20260927 22:51:52.190639 19096 ts_tablet_manager.cc:1397] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260927 22:51:52.191638 19099 tablet_bootstrap.cc:654] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:52.192265 19097 tablet_bootstrap.cc:492] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: No bootstrap required, opened a new log
I20260927 22:51:52.192498 19097 ts_tablet_manager.cc:1397] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260927 22:51:52.192732 19099 log.cc:824] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:52.195582 19099 tablet_bootstrap.cc:492] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: No bootstrap required, opened a new log
I20260927 22:51:52.195818 19099 ts_tablet_manager.cc:1397] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 22:51:52.200050 19097 raft_consensus.cc:348] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.200428 19097 raft_consensus.cc:374] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:52.200526 19097 raft_consensus.cc:729] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6e19c1c27e487abab46986729ac551, State: Initialized, Role: FOLLOWER
I20260927 22:51:52.200529 19096 raft_consensus.cc:348] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.200924 19096 raft_consensus.cc:374] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:52.201017 19096 raft_consensus.cc:729] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658d69fbd6c9455c966acea5a2e5d745, State: Initialized, Role: FOLLOWER
I20260927 22:51:52.201092 19097 consensus_queue.cc:260] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.201571 19096 consensus_queue.cc:260] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.202697 19097 ts_tablet_manager.cc:1428] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 22:51:52.203879 18945 heartbeater.cc:503] Master 127.0.97.62:40973 was elected leader, sending a full tablet report...
I20260927 22:51:52.204449 19096 ts_tablet_manager.cc:1428] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 22:51:52.205387 19099 raft_consensus.cc:348] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.205698 19099 raft_consensus.cc:374] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:52.205789 19099 raft_consensus.cc:729] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e66a354b4f403aa1a91ef8c0d93a7b, State: Initialized, Role: FOLLOWER
I20260927 22:51:52.206314 19099 consensus_queue.cc:260] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.207284 19078 heartbeater.cc:503] Master 127.0.97.62:40973 was elected leader, sending a full tablet report...
I20260927 22:51:52.207898 19099 ts_tablet_manager.cc:1428] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260927 22:51:52.327713 19079 tablet.cc:2369] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:52.345252 18946 tablet.cc:2369] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:52.362258 18812 tablet.cc:2369] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:52.402067 19104 raft_consensus.cc:482] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:52.402355 19104 raft_consensus.cc:504] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.404072 19104 leader_election.cc:288] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf6e19c1c27e487abab46986729ac551 (127.0.97.1:44045), 658d69fbd6c9455c966acea5a2e5d745 (127.0.97.2:43275)
I20260927 22:51:52.409246 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.404419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43275 (local address 127.0.97.3:38479)
0927 22:51:52.404604 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:52.404623 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:52.404749 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:52.405095 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:52.405100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:52.405130 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:52.405294 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.405302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.406242 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.406245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.407528 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.407539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.408582 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.408587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.408749 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.408772 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:51:52.408851 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:51:52.408977 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:51:52.409246 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.404560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38479 (local address 127.0.97.2:43275)
0927 22:51:52.404739 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.404744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.404765 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:52.404892 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.404895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.404962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.405036 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:52.405434 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.406111 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.407675 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.408431 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.408550 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.408643 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.408895 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.409023 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:51:52.410211 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.404438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44045 (local address 127.0.97.3:60735)
0927 22:51:52.405325 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:52.405339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:52.405461 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:52.405768 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:52.405772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:52.405795 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:52.406081 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.406089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.407108 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.407111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.408432 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:52.408443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.409575 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.409582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:52.409741 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.409774 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:51:52.409856 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 22:51:52.409957 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":151,"thread_start_us":393,"threads_started":1}
I20260927 22:51:52.410279 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.404493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60735 (local address 127.0.97.1:44045)
0927 22:51:52.404687 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.404693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.405439 (+ 746us) server_negotiation.cc:407] Connection header received
0927 22:51:52.405600 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.405604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.405662 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.405733 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:52.406229 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.406958 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.408587 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.409334 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.409471 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.409558 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.409911 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.410040 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:52.410393 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e6336a728c45869a18aace0ef5585a" candidate_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658d69fbd6c9455c966acea5a2e5d745" is_pre_election: true
I20260927 22:51:52.410979 18900 raft_consensus.cc:2454] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8e66a354b4f403aa1a91ef8c0d93a7b in term 0.
I20260927 22:51:52.411265 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e6336a728c45869a18aace0ef5585a" candidate_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6e19c1c27e487abab46986729ac551" is_pre_election: true
I20260927 22:51:52.411733 18969 leader_election.cc:302] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658d69fbd6c9455c966acea5a2e5d745, e8e66a354b4f403aa1a91ef8c0d93a7b; no voters:
I20260927 22:51:52.411813 18766 raft_consensus.cc:2454] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8e66a354b4f403aa1a91ef8c0d93a7b in term 0.
I20260927 22:51:52.412137 19104 raft_consensus.cc:2781] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:52.412312 19104 raft_consensus.cc:482] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:52.412459 19104 raft_consensus.cc:3037] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:52.414731 19104 raft_consensus.cc:504] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.415284 19104 leader_election.cc:288] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [CANDIDATE]: Term 1 election: Requested vote from peers cf6e19c1c27e487abab46986729ac551 (127.0.97.1:44045), 658d69fbd6c9455c966acea5a2e5d745 (127.0.97.2:43275)
I20260927 22:51:52.415752 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e6336a728c45869a18aace0ef5585a" candidate_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6e19c1c27e487abab46986729ac551"
I20260927 22:51:52.415822 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e6336a728c45869a18aace0ef5585a" candidate_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658d69fbd6c9455c966acea5a2e5d745"
I20260927 22:51:52.415934 18766 raft_consensus.cc:3037] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:52.416039 18900 raft_consensus.cc:3037] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:52.418097 18766 raft_consensus.cc:2454] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e66a354b4f403aa1a91ef8c0d93a7b in term 1.
I20260927 22:51:52.418232 18900 raft_consensus.cc:2454] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e66a354b4f403aa1a91ef8c0d93a7b in term 1.
I20260927 22:51:52.418525 18968 leader_election.cc:302] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6e19c1c27e487abab46986729ac551, e8e66a354b4f403aa1a91ef8c0d93a7b; no voters:
I20260927 22:51:52.418782 19104 raft_consensus.cc:2781] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:52.419183 19104 raft_consensus.cc:686] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 1 LEADER]: Becoming Leader. State: Replica: e8e66a354b4f403aa1a91ef8c0d93a7b, State: Running, Role: LEADER
I20260927 22:51:52.419683 19104 consensus_queue.cc:237] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } }
I20260927 22:51:52.424302 18621 catalog_manager.cc:5495] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b reported cstate change: term changed from 0 to 1, leader changed from <none> to e8e66a354b4f403aa1a91ef8c0d93a7b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8e66a354b4f403aa1a91ef8c0d93a7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:52.454321 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.449285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.97.3:41743)
0927 22:51:52.449460 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.449464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.449483 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:51:52.449654 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.449658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.449740 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.449870 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:52.450208 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.450933 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.452369 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.452767 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.452866 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.453340 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:52.453839 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:52.453987 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.454080 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.454124 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:51:52.462291 18900 raft_consensus.cc:1261] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 1 FOLLOWER]: Refusing update from remote peer e8e66a354b4f403aa1a91ef8c0d93a7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:52.462311 18766 raft_consensus.cc:1261] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 1 FOLLOWER]: Refusing update from remote peer e8e66a354b4f403aa1a91ef8c0d93a7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:52.463189 19106 consensus_queue.cc:1035] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "658d69fbd6c9455c966acea5a2e5d745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:52.463551 19104 consensus_queue.cc:1035] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6e19c1c27e487abab46986729ac551" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:52.473835 19109 mvcc.cc:205] Tried to move back new op lower bound from 7334090803035447296 to 7334090802874486784. Current Snapshot: MvccSnapshot[applied={T|T < 7334090803035447296}]
I20260927 22:51:52.487733 18621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:52.487998 18621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:52.491636 18621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaed8c4c553e4e90b26cf04e09162166: Sending DeleteTablet for 3 replicas of tablet 16e6336a728c45869a18aace0ef5585a
I20260927 22:51:52.492952 19013 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e6336a728c45869a18aace0ef5585a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38502
I20260927 22:51:52.492956 18746 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e6336a728c45869a18aace0ef5585a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33496
I20260927 22:51:52.493268 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e6336a728c45869a18aace0ef5585a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48834
I20260927 22:51:52.497066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18683
I20260927 22:51:52.506457 19118 tablet_replica.cc:330] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: stopping tablet replica
I20260927 22:51:52.507155 19118 raft_consensus.cc:2229] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:52.507891 19118 raft_consensus.cc:2258] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:52.507920 19120 tablet_replica.cc:330] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: stopping tablet replica
I20260927 22:51:52.508311 19120 raft_consensus.cc:2229] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:52.508780 19120 raft_consensus.cc:2258] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:52.511251 19120 ts_tablet_manager.cc:1905] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:51:52 dist-test-slave-kr25 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:52.519097 19120 ts_tablet_manager.cc:1918] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:52.519276 19120 log.cc:1196] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/16e6336a728c45869a18aace0ef5585a
I20260927 22:51:52.519273 19118 ts_tablet_manager.cc:1905] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:52.519829 19120 ts_tablet_manager.cc:1939] T 16e6336a728c45869a18aace0ef5585a P 658d69fbd6c9455c966acea5a2e5d745: Deleting consensus metadata
I20260927 22:51:52.520125 19119 tablet_replica.cc:330] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: stopping tablet replica
I20260927 22:51:52.521739 18608 catalog_manager.cc:4841] TS 658d69fbd6c9455c966acea5a2e5d745 (127.0.97.2:43275): tablet 16e6336a728c45869a18aace0ef5585a (table test-table [id=92afac30add64ea9bcd96583137bb40b]) successfully deleted
I20260927 22:51:52.522329 19119 raft_consensus.cc:2229] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:52.523046 19119 raft_consensus.cc:2258] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:52.525414 19119 ts_tablet_manager.cc:1905] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:52.526355 19118 ts_tablet_manager.cc:1918] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:52.526526 19118 log.cc:1196] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/16e6336a728c45869a18aace0ef5585a
I20260927 22:51:52.527016 19118 ts_tablet_manager.cc:1939] T 16e6336a728c45869a18aace0ef5585a P cf6e19c1c27e487abab46986729ac551: Deleting consensus metadata
I20260927 22:51:52.527660 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.500343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.0.97.1:44045)
0927 22:51:52.500549 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.500556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.502015 (+ 1459us) server_negotiation.cc:407] Connection header received
0927 22:51:52.502226 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.502232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.502342 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.502459 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.509683 (+ 7224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.510802 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.512756 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.513378 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.513481 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.520129 (+ 6648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.520145 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.520165 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.520209 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.523984 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.524920 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.524930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.524934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.525121 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.525605 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.525611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.525614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.526075 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.526377 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.526972 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.527231 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:51:52.529174 18806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:52.529135 18607 catalog_manager.cc:4841] TS cf6e19c1c27e487abab46986729ac551 (127.0.97.1:44045): tablet 16e6336a728c45869a18aace0ef5585a (table test-table [id=92afac30add64ea9bcd96583137bb40b]) successfully deleted
I20260927 22:51:52.533949 19119 ts_tablet_manager.cc:1918] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:52.534142 19119 log.cc:1196] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/16e6336a728c45869a18aace0ef5585a
I20260927 22:51:52.534590 19119 ts_tablet_manager.cc:1939] T 16e6336a728c45869a18aace0ef5585a P e8e66a354b4f403aa1a91ef8c0d93a7b: Deleting consensus metadata
I20260927 22:51:52.536245 18607 catalog_manager.cc:4841] TS e8e66a354b4f403aa1a91ef8c0d93a7b (127.0.97.3:41743): tablet 16e6336a728c45869a18aace0ef5585a (table test-table [id=92afac30add64ea9bcd96583137bb40b]) successfully deleted
I20260927 22:51:52.679390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18683
I20260927 22:51:52.693687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18817
Sep 27 22:51:52 dist-test-slave-kr25 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:52.710345 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.694467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.0.97.2:43275)
0927 22:51:52.694689 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.694696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.694775 (+ 79us) server_negotiation.cc:407] Connection header received
0927 22:51:52.694943 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.694949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.695054 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.695169 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.696662 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.697785 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.699139 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.699847 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.699981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.704029 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.704049 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.704071 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.704115 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.707743 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.708375 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.708382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.708385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.708528 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.708870 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.708875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.708879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.709248 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.709516 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.709810 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.710033 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:51:52.711009 18940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:52.800102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18817
I20260927 22:51:52.820186 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18950
Sep 27 22:51:52 dist-test-slave-kr25 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:52.835609 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.820943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.0.97.3:41743)
0927 22:51:52.821142 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.821148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.821168 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:51:52.821247 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.821252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.821320 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.821420 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.822762 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.823719 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.825033 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.825484 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.825588 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.829961 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.829972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.829986 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.830022 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.832875 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.833515 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.833522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.833524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.833673 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.834036 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.834041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.834045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.834370 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.834695 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.834977 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.835171 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:51:52.836295 19073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:52.953418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18950
I20260927 22:51:52.968510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 18589
I20260927 22:51:52.969542 18651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:53.075842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 18589
2026-09-27T22:51:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4165 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T22:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:53Z Disabled control of system clock
I20260927 22:51:53.144194 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40035
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:51:53.476468 19129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:53.476734 19129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:53.476852 19129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:53.484123 19129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:53.484239 19129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:53.484305 19129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:53.484386 19129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:53.484452 19129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:53.494179 19129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:53.495599 19129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:53.497440 19129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:53.509112 19134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:53.509048 19137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:53.509048 19135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:53.509812 19129 server_base.cc:1034] running on GCE node
I20260927 22:51:53.510550 19129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:53.511950 19129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:53.513166 19129 hybrid_clock.cc:648] HybridClock initialized: now 1790549513513108 us; error 37 us; skew 500 ppm
I20260927 22:51:53.513597 19129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:53.516333 19129 webserver.cc:460] Webserver started at http://127.0.97.62:36037/ using document root <none> and password file <none>
I20260927 22:51:53.517141 19129 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:53.517246 19129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:53.517575 19129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:53.521283 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "de5c98ed52d34466879b934273e292d5"
format_stamp: "Formatted at 2026-09-27 22:51:53 on dist-test-slave-kr25"
server_key: "1f585f603267f9db84b073a8cae2def7"
server_key_iv: "f1f8f0dc350c37915bfcd8694a8d5c76"
server_key_version: "encryptionkey@0"
I20260927 22:51:53.522074 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "de5c98ed52d34466879b934273e292d5"
format_stamp: "Formatted at 2026-09-27 22:51:53 on dist-test-slave-kr25"
server_key: "1f585f603267f9db84b073a8cae2def7"
server_key_iv: "f1f8f0dc350c37915bfcd8694a8d5c76"
server_key_version: "encryptionkey@0"
I20260927 22:51:53.527210 19129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:51:53.530759 19143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:53.532594 19129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:53.532800 19129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "de5c98ed52d34466879b934273e292d5"
format_stamp: "Formatted at 2026-09-27 22:51:53 on dist-test-slave-kr25"
server_key: "1f585f603267f9db84b073a8cae2def7"
server_key_iv: "f1f8f0dc350c37915bfcd8694a8d5c76"
server_key_version: "encryptionkey@0"
I20260927 22:51:53.533027 19129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:53.559397 19129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:53.560256 19129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:53.560508 19129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:53.577945 19129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40035
I20260927 22:51:53.577970 19194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40035 every 8 connection(s)
I20260927 22:51:53.579272 19129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:53.582955 19195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:53.585170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 19129
I20260927 22:51:53.585318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:53.585738 388 external_mini_cluster.cc:1428] Setting key 3572754a184dd3f1ae9a5982e0c8f4dd
I20260927 22:51:53.595573 19195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5: Bootstrap starting.
I20260927 22:51:53.599351 19195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:53.600652 19195 log.cc:824] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:53.601025 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:53.587653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.0.97.62:40035)
0927 22:51:53.588738 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:53.588753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:53.588801 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:51:53.591015 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:53.591040 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:53.591646 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:53.592071 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:53.592619 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:53.594579 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:53.595964 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:53.596962 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:53.597166 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:53.598201 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:53.598210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:53.598226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:53.598909 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:53.599053 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:53.599080 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:53.599204 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:53.599466 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 22:51:53.599813 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":338,"threads_started":1}
I20260927 22:51:53.604542 19195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5: No bootstrap required, opened a new log
I20260927 22:51:53.611495 19195 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c98ed52d34466879b934273e292d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40035 } }
I20260927 22:51:53.611889 19195 raft_consensus.cc:374] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:53.611966 19195 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5c98ed52d34466879b934273e292d5, State: Initialized, Role: FOLLOWER
I20260927 22:51:53.612483 19195 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c98ed52d34466879b934273e292d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40035 } }
I20260927 22:51:53.612684 19195 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:53.612779 19195 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:53.612914 19195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:53.615108 19195 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c98ed52d34466879b934273e292d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40035 } }
I20260927 22:51:53.615641 19195 leader_election.cc:302] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5c98ed52d34466879b934273e292d5; no voters:
I20260927 22:51:53.616178 19195 leader_election.cc:288] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:53.616331 19200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:53.617027 19200 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [term 1 LEADER]: Becoming Leader. State: Replica: de5c98ed52d34466879b934273e292d5, State: Running, Role: LEADER
I20260927 22:51:53.617561 19200 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c98ed52d34466879b934273e292d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40035 } }
I20260927 22:51:53.618233 19195 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:53.620493 19202 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5c98ed52d34466879b934273e292d5. Latest consensus state: current_term: 1 leader_uuid: "de5c98ed52d34466879b934273e292d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c98ed52d34466879b934273e292d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40035 } } }
I20260927 22:51:53.620667 19201 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5c98ed52d34466879b934273e292d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c98ed52d34466879b934273e292d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40035 } } }
I20260927 22:51:53.620920 19202 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:53.620961 19201 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:53.624478 19206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:53.629222 19206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:53.638233 19206 catalog_manager.cc:1347] Generated new cluster ID: 075c13d6b7e3469885d036ea08ca94e0
I20260927 22:51:53.638355 19206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:53.670694 19206 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:53.672022 19206 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:53.680195 19206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5: Generated new TSK 0
I20260927 22:51:53.680974 19206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:51:53.690354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40035
--builtin_ntp_servers=127.0.97.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:51:54.032837 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:54.033097 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:54.033202 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:54.040431 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:54.040526 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:54.040671 19219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:54.049643 19219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:54.050913 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:54.052742 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:54.065476 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.065546 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.065515 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:54.066368 19219 server_base.cc:1034] running on GCE node
I20260927 22:51:54.066954 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:54.067930 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:54.069157 19219 hybrid_clock.cc:648] HybridClock initialized: now 1790549514069101 us; error 43 us; skew 500 ppm
I20260927 22:51:54.069458 19219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:54.071539 19219 webserver.cc:460] Webserver started at http://127.0.97.1:39277/ using document root <none> and password file <none>
I20260927 22:51:54.072172 19219 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:54.072268 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:54.072571 19219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:54.075270 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "ec51f8251413404b958865166d7f2328"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "d4fb6976e4a32986a149680dbd68ada9"
server_key_iv: "9087491bce69dee9ad09b02025399fe2"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.076069 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ec51f8251413404b958865166d7f2328"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "d4fb6976e4a32986a149680dbd68ada9"
server_key_iv: "9087491bce69dee9ad09b02025399fe2"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.080822 19219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:51:54.083984 19233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.085171 19219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 22:51:54.085314 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "ec51f8251413404b958865166d7f2328"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "d4fb6976e4a32986a149680dbd68ada9"
server_key_iv: "9087491bce69dee9ad09b02025399fe2"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.085516 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:54.121658 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:54.122572 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:54.122849 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:54.123862 19219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:54.125306 19219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:54.125399 19219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.125511 19219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:54.125578 19219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.154603 19219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39619
I20260927 22:51:54.154656 19345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39619 every 8 connection(s)
I20260927 22:51:54.155870 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:54.156764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 19219
I20260927 22:51:54.156929 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:54.157369 388 external_mini_cluster.cc:1428] Setting key fed1435cce8903ac8b63422797428783
I20260927 22:51:54.160339 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40035
--builtin_ntp_servers=127.0.97.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:51:54.168285 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:54.158383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40975 (local address 127.0.97.62:40035)
0927 22:51:54.158836 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:54.158842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:54.160198 (+ 1356us) server_negotiation.cc:407] Connection header received
0927 22:51:54.162023 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:54.162029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:54.162110 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:54.162205 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:54.163269 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.164059 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.165583 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.165994 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.166103 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:54.167417 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:54.167424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:54.167429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:54.167480 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:54.167527 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:54.167540 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:54.167610 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:54.167823 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 22:51:54.167936 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260927 22:51:54.169277 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:54.159098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40035 (local address 127.0.97.1:40975)
0927 22:51:54.160053 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:54.160091 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:54.161763 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:54.162451 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:54.162460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:54.162556 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:54.163109 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:54.163126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.164209 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.164213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:54.165447 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:54.165458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.166204 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.166212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:54.166450 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:54.167111 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:54.167113 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:54.167155 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:54.167236 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:54.167632 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:54.167642 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:54.167649 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:51:54.167803 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:51:54.168210 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":328,"threads_started":1}
I20260927 22:51:54.170823 19346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40035
I20260927 22:51:54.171129 19346 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:54.171942 19346 heartbeater.cc:511] Master 127.0.97.62:40035 requested a full tablet report, sending...
I20260927 22:51:54.174336 19160 ts_manager.cc:194] Registered new tserver with Master: ec51f8251413404b958865166d7f2328 (127.0.97.1:39619)
I20260927 22:51:54.176445 19160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40975
W20260927 22:51:54.472971 19350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:54.473228 19350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:54.473322 19350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:54.480441 19350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:54.480553 19350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:54.480660 19350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:54.489580 19350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:54.490828 19350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:54.492599 19350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:54.505177 19358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.505139 19356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.505210 19355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:54.506125 19350 server_base.cc:1034] running on GCE node
I20260927 22:51:54.506678 19350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:54.507613 19350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:54.508839 19350 hybrid_clock.cc:648] HybridClock initialized: now 1790549514508783 us; error 44 us; skew 500 ppm
I20260927 22:51:54.509157 19350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:54.511291 19350 webserver.cc:460] Webserver started at http://127.0.97.2:44535/ using document root <none> and password file <none>
I20260927 22:51:54.511958 19350 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:54.512055 19350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:54.512346 19350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:54.515022 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd8b78092dd1453e8479df821eaf1025"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "95afb00820ea276f014eaf7def7b800d"
server_key_iv: "3e4ca5bc9fa2914a2d47dcf490f838bd"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.515777 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd8b78092dd1453e8479df821eaf1025"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "95afb00820ea276f014eaf7def7b800d"
server_key_iv: "3e4ca5bc9fa2914a2d47dcf490f838bd"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.520498 19350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 22:51:54.523725 19364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.524988 19350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:51:54.525126 19350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "bd8b78092dd1453e8479df821eaf1025"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "95afb00820ea276f014eaf7def7b800d"
server_key_iv: "3e4ca5bc9fa2914a2d47dcf490f838bd"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.525311 19350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:54.542389 19350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:54.543226 19350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:54.543526 19350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:54.544480 19350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:54.545884 19350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:54.545980 19350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.546087 19350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:54.546154 19350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.575253 19350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40407
I20260927 22:51:54.575302 19476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40407 every 8 connection(s)
I20260927 22:51:54.576538 19350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:54.582798 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 19350
I20260927 22:51:54.582957 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:54.583488 388 external_mini_cluster.cc:1428] Setting key bf859a220ac00d452b648557c551aa27
I20260927 22:51:54.586412 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40035
--builtin_ntp_servers=127.0.97.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:51:54.589871 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:54.578978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46965 (local address 127.0.97.62:40035)
0927 22:51:54.579195 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:54.579202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:54.580815 (+ 1613us) server_negotiation.cc:407] Connection header received
0927 22:51:54.583195 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:54.583202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:54.583284 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:54.583411 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:54.584657 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.585592 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.587152 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.587597 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.587707 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:54.589080 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:54.589085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:54.589088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:54.589104 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:54.589155 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:54.589169 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:54.589240 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:54.589477 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:51:54.589567 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:51:54.590942 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:54.579645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40035 (local address 127.0.97.2:46965)
0927 22:51:54.580662 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:54.580704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:54.582919 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:54.583668 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:54.583680 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:54.583791 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:54.584500 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:54.584522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.585729 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.585734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:54.587023 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:54.587034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.587932 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.587937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:54.588133 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:54.588768 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:54.588771 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:54.588843 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:54.588924 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:54.589249 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:54.589260 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:54.589268 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:51:54.589463 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 22:51:54.589834 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
I20260927 22:51:54.592203 19477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40035
I20260927 22:51:54.592475 19477 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:54.593169 19477 heartbeater.cc:511] Master 127.0.97.62:40035 requested a full tablet report, sending...
I20260927 22:51:54.594420 19160 ts_manager.cc:194] Registered new tserver with Master: bd8b78092dd1453e8479df821eaf1025 (127.0.97.2:40407)
I20260927 22:51:54.595188 19160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:46965
W20260927 22:51:54.900015 19480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:54.900295 19480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:54.900403 19480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:54.907533 19480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:54.907625 19480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:54.907764 19480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:54.916730 19480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:54.917968 19480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:54.919744 19480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:54.932231 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.932185 19488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.932265 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:54.932948 19480 server_base.cc:1034] running on GCE node
I20260927 22:51:54.933516 19480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:54.934473 19480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:54.935688 19480 hybrid_clock.cc:648] HybridClock initialized: now 1790549514935630 us; error 54 us; skew 500 ppm
I20260927 22:51:54.935992 19480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:54.938135 19480 webserver.cc:460] Webserver started at http://127.0.97.3:46523/ using document root <none> and password file <none>
I20260927 22:51:54.938840 19480 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:54.938944 19480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:54.939239 19480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:54.942094 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "74b6f29275a0449eb18312be6b777e7c"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "51ff819c658a382e41fdc5432f13b597"
server_key_iv: "7e175c5a1379c7fd38a77364e4d75209"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.942828 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "74b6f29275a0449eb18312be6b777e7c"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "51ff819c658a382e41fdc5432f13b597"
server_key_iv: "7e175c5a1379c7fd38a77364e4d75209"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.947606 19480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 22:51:54.950762 19494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.951988 19480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:54.952121 19480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "74b6f29275a0449eb18312be6b777e7c"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-kr25"
server_key: "51ff819c658a382e41fdc5432f13b597"
server_key_iv: "7e175c5a1379c7fd38a77364e4d75209"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.952299 19480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:54.982666 19480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:54.983527 19480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:54.983779 19480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:54.984714 19480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:54.986138 19480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:54.986235 19480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.986338 19480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:54.986407 19480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:55.015652 19480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44591
I20260927 22:51:55.015705 19606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44591 every 8 connection(s)
I20260927 22:51:55.016947 19480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:55.021337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 19480
I20260927 22:51:55.021510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:55.022101 388 external_mini_cluster.cc:1428] Setting key 7bd5abb64fa012046bd7ef6905399fbd
I20260927 22:51:55.029544 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.019353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44505 (local address 127.0.97.62:40035)
0927 22:51:55.019541 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.019545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.020982 (+ 1437us) server_negotiation.cc:407] Connection header received
0927 22:51:55.023264 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.023270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.023353 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.023466 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:55.024648 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.025426 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.026910 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.027321 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.027425 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.028760 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:55.028765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:55.028768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:55.028786 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:55.028815 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:55.028826 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:55.028927 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.029109 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.029200 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:51:55.030515 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.019975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40035 (local address 127.0.97.3:44505)
0927 22:51:55.020848 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.020883 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.022998 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.023709 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.023718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.023816 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:55.024493 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.024512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.025560 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.025564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.026778 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.026789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.027547 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.027555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.027772 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.028467 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:55.028470 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:55.028508 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:55.028584 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:55.028907 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:55.028917 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:55.028925 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.029073 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.029473 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":260,"threads_started":1}
I20260927 22:51:55.031772 19607 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40035
I20260927 22:51:55.032058 19607 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:55.032783 19607 heartbeater.cc:511] Master 127.0.97.62:40035 requested a full tablet report, sending...
I20260927 22:51:55.034056 19160 ts_manager.cc:194] Registered new tserver with Master: 74b6f29275a0449eb18312be6b777e7c (127.0.97.3:44591)
I20260927 22:51:55.034781 19160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:44505
I20260927 22:51:55.037067 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:55.045984 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.040823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.0.97.62:40035)
0927 22:51:55.041057 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.041063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.041347 (+ 284us) server_negotiation.cc:407] Connection header received
0927 22:51:55.041465 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.041468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.041537 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.041614 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:55.042005 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.042670 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.044025 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.044399 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.044485 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.045359 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:55.045364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:55.045367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:55.045383 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:55.045412 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:55.045423 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:55.045525 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.045637 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.045680 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:51:55.049306 19160 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:55.056172 19160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:55.080729 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.067957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.97.1:39619)
0927 22:51:55.072789 (+ 4832us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.072797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.072823 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:51:55.072934 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.072943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.073200 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.073400 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:55.073939 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.075707 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.078217 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.079576 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.079741 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.080225 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.080293 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.080443 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4693,"thread_start_us":4608,"threads_started":1}
I20260927 22:51:55.080753 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.067397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:44984)
0927 22:51:55.070702 (+ 3305us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.070719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.070826 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.073406 (+ 2580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.073412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.073439 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:55.073772 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.073781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.075824 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.075828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.078021 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.078041 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.079797 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.079803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.080000 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.080051 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.080169 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.080359 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3196,"mutex_wait_us":18,"thread_start_us":4644,"threads_started":1}
I20260927 22:51:55.081509 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.067322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44591 (local address 127.0.0.1:37902)
0927 22:51:55.069392 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.069439 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.069604 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.074350 (+ 4746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.074355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.074377 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:55.074726 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.074737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.076560 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.076566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.078199 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.078213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.080340 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.080346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.080534 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.080566 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.080646 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.080781 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"mutex_wait_us":23,"thread_start_us":2464,"threads_started":1}
I20260927 22:51:55.081585 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.068196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.0.97.2:40407)
0927 22:51:55.073529 (+ 5333us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.073534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.073557 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:51:55.073726 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.073731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.073891 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.074053 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:55.074668 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.076250 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.078856 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.080232 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.080353 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.080584 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.081080 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.081238 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5180,"thread_start_us":5060,"threads_started":1}
I20260927 22:51:55.081548 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.068414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40407 (local address 127.0.0.1:56344)
0927 22:51:55.072259 (+ 3845us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.072277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.072384 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.074055 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.074060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.074082 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:55.074455 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.074470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.076389 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.076394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.078665 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.078679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.080714 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.080720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.080905 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.080935 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.081020 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.081129 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3739,"mutex_wait_us":86,"thread_start_us":2064,"threads_started":1}
I20260927 22:51:55.082309 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.067482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.0.97.3:44591)
0927 22:51:55.073218 (+ 5736us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.073225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.073255 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:51:55.073953 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.073961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.074173 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.074358 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:55.074953 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.076434 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.078397 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.079825 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.079954 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.080212 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.081862 (+ 1650us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.082081 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5589,"thread_start_us":5458,"threads_started":1}
I20260927 22:51:55.084743 19281 tablet_service.cc:1467] Processing CreateTablet for tablet ba1d9cfa64dd454a9a93f5496a759a7e (DEFAULT_TABLE table=test-table [id=92c370898ae645b6a766b2d3126833fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:55.085630 19542 tablet_service.cc:1467] Processing CreateTablet for tablet ba1d9cfa64dd454a9a93f5496a759a7e (DEFAULT_TABLE table=test-table [id=92c370898ae645b6a766b2d3126833fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:55.086201 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1d9cfa64dd454a9a93f5496a759a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:55.086160 19412 tablet_service.cc:1467] Processing CreateTablet for tablet ba1d9cfa64dd454a9a93f5496a759a7e (DEFAULT_TABLE table=test-table [id=92c370898ae645b6a766b2d3126833fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:55.086930 19542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1d9cfa64dd454a9a93f5496a759a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:55.087200 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1d9cfa64dd454a9a93f5496a759a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:55.098977 19625 tablet_bootstrap.cc:492] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Bootstrap starting.
I20260927 22:51:55.101125 19626 tablet_bootstrap.cc:492] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: Bootstrap starting.
I20260927 22:51:55.102835 19625 tablet_bootstrap.cc:654] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:55.103248 19627 tablet_bootstrap.cc:492] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Bootstrap starting.
I20260927 22:51:55.104300 19625 log.cc:824] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:55.104975 19626 tablet_bootstrap.cc:654] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:55.106274 19627 tablet_bootstrap.cc:654] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:55.106336 19626 log.cc:824] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:55.107419 19625 tablet_bootstrap.cc:492] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: No bootstrap required, opened a new log
I20260927 22:51:55.107468 19627 log.cc:824] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:55.107728 19625 ts_tablet_manager.cc:1397] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260927 22:51:55.113746 19627 tablet_bootstrap.cc:492] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: No bootstrap required, opened a new log
I20260927 22:51:55.113981 19627 ts_tablet_manager.cc:1397] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260927 22:51:55.114612 19626 tablet_bootstrap.cc:492] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: No bootstrap required, opened a new log
I20260927 22:51:55.114862 19626 ts_tablet_manager.cc:1397] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260927 22:51:55.117579 19625 raft_consensus.cc:348] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.118124 19625 raft_consensus.cc:374] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:55.118223 19625 raft_consensus.cc:729] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8b78092dd1453e8479df821eaf1025, State: Initialized, Role: FOLLOWER
I20260927 22:51:55.118878 19625 consensus_queue.cc:260] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.119967 19477 heartbeater.cc:503] Master 127.0.97.62:40035 was elected leader, sending a full tablet report...
I20260927 22:51:55.121614 19627 raft_consensus.cc:348] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.121929 19625 ts_tablet_manager.cc:1428] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260927 22:51:55.122120 19627 raft_consensus.cc:374] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:55.122231 19627 raft_consensus.cc:729] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b6f29275a0449eb18312be6b777e7c, State: Initialized, Role: FOLLOWER
I20260927 22:51:55.122862 19627 consensus_queue.cc:260] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.123947 19607 heartbeater.cc:503] Master 127.0.97.62:40035 was elected leader, sending a full tablet report...
I20260927 22:51:55.124644 19627 ts_tablet_manager.cc:1428] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260927 22:51:55.124394 19626 raft_consensus.cc:348] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.124786 19626 raft_consensus.cc:374] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:55.124869 19626 raft_consensus.cc:729] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec51f8251413404b958865166d7f2328, State: Initialized, Role: FOLLOWER
I20260927 22:51:55.125464 19626 consensus_queue.cc:260] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.126986 19346 heartbeater.cc:503] Master 127.0.97.62:40035 was elected leader, sending a full tablet report...
I20260927 22:51:55.127648 19626 ts_tablet_manager.cc:1428] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260927 22:51:55.157959 19347 tablet.cc:2369] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:55.268509 19608 tablet.cc:2369] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:55.328476 19478 tablet.cc:2369] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:55.519842 19632 raft_consensus.cc:482] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:55.520195 19632 raft_consensus.cc:504] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.521835 19632 leader_election.cc:288] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec51f8251413404b958865166d7f2328 (127.0.97.1:39619), bd8b78092dd1453e8479df821eaf1025 (127.0.97.2:40407)
I20260927 22:51:55.528255 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.522150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60095 (local address 127.0.97.1:39619)
0927 22:51:55.522363 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.522369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.522389 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:51:55.522634 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.522640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.522731 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.522809 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:55.523259 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.524236 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.526425 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.527469 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.527596 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.527689 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.527878 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.528009 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:51:55.528281 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.522015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.97.3:60095)
0927 22:51:55.522302 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.522320 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.522440 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.522879 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.522884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.522911 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:55.523111 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.523122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.524370 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.524373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.526272 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.526287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.527606 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.527610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.527741 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.527765 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.527836 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.527940 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":19}
I20260927 22:51:55.528739 19633 raft_consensus.cc:482] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:55.528891 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.522429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53059 (local address 127.0.97.2:40407)
0927 22:51:55.522609 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.522614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.522902 (+ 288us) server_negotiation.cc:407] Connection header received
0927 22:51:55.523166 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.523172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.523243 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.523344 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:55.523809 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.524649 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.526776 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.527908 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.528018 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.528103 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.528523 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.528666 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:51:55.529289 19301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1d9cfa64dd454a9a93f5496a759a7e" candidate_uuid: "74b6f29275a0449eb18312be6b777e7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec51f8251413404b958865166d7f2328" is_pre_election: true
I20260927 22:51:55.529529 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.522209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40407 (local address 127.0.97.3:53059)
0927 22:51:55.522796 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.522812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.522973 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.523364 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.523367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.523388 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:55.523678 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.523686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.524789 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.524794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.526624 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.526634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.528063 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.528069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.528319 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.528349 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.529042 (+ 693us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.529169 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":14,"spinlock_wait_cycles":3968,"thread_start_us":306,"threads_started":1}
I20260927 22:51:55.529263 19633 raft_consensus.cc:504] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.530174 19301 raft_consensus.cc:2454] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b6f29275a0449eb18312be6b777e7c in term 0.
I20260927 22:51:55.530562 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1d9cfa64dd454a9a93f5496a759a7e" candidate_uuid: "74b6f29275a0449eb18312be6b777e7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8b78092dd1453e8479df821eaf1025" is_pre_election: true
I20260927 22:51:55.530925 19498 leader_election.cc:302] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b6f29275a0449eb18312be6b777e7c, ec51f8251413404b958865166d7f2328; no voters:
I20260927 22:51:55.531134 19432 raft_consensus.cc:2454] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b6f29275a0449eb18312be6b777e7c in term 0.
I20260927 22:51:55.531324 19632 raft_consensus.cc:2781] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:55.531512 19632 raft_consensus.cc:482] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:55.531615 19632 raft_consensus.cc:3037] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:55.534806 19632 raft_consensus.cc:504] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.535413 19632 leader_election.cc:288] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [CANDIDATE]: Term 1 election: Requested vote from peers ec51f8251413404b958865166d7f2328 (127.0.97.1:39619), bd8b78092dd1453e8479df821eaf1025 (127.0.97.2:40407)
I20260927 22:51:55.535882 19301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1d9cfa64dd454a9a93f5496a759a7e" candidate_uuid: "74b6f29275a0449eb18312be6b777e7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec51f8251413404b958865166d7f2328"
I20260927 22:51:55.536015 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1d9cfa64dd454a9a93f5496a759a7e" candidate_uuid: "74b6f29275a0449eb18312be6b777e7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8b78092dd1453e8479df821eaf1025"
I20260927 22:51:55.536121 19301 raft_consensus.cc:3037] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:55.536209 19432 raft_consensus.cc:3037] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:55.538538 19633 leader_election.cc:288] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74b6f29275a0449eb18312be6b777e7c (127.0.97.3:44591), bd8b78092dd1453e8479df821eaf1025 (127.0.97.2:40407)
I20260927 22:51:55.539345 19432 raft_consensus.cc:2454] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b6f29275a0449eb18312be6b777e7c in term 1.
I20260927 22:51:55.539496 19301 raft_consensus.cc:2454] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b6f29275a0449eb18312be6b777e7c in term 1.
I20260927 22:51:55.539870 19495 leader_election.cc:302] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b6f29275a0449eb18312be6b777e7c, bd8b78092dd1453e8479df821eaf1025; no voters:
I20260927 22:51:55.540373 19632 raft_consensus.cc:2781] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:55.540740 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.531926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54111 (local address 127.0.97.3:44591)
0927 22:51:55.532111 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.532117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.532485 (+ 368us) server_negotiation.cc:407] Connection header received
0927 22:51:55.532665 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.532671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.532748 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.532843 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:55.533443 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.534509 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.538133 (+ 3624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.539381 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.539564 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.539674 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.540292 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.540446 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:51:55.540850 19632 raft_consensus.cc:686] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 1 LEADER]: Becoming Leader. State: Replica: 74b6f29275a0449eb18312be6b777e7c, State: Running, Role: LEADER
I20260927 22:51:55.541354 19632 consensus_queue.cc:237] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } }
I20260927 22:51:55.541640 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.531514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44591 (local address 127.0.97.1:54111)
0927 22:51:55.532373 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.532400 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.532537 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.532884 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.532889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.532918 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:55.533290 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.533303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.534690 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.534697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.537794 (+ 3097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.537902 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.539871 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.539878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.540086 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.540122 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.541134 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.541294 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":553,"threads_started":1}
I20260927 22:51:55.542529 19562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1d9cfa64dd454a9a93f5496a759a7e" candidate_uuid: "ec51f8251413404b958865166d7f2328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b6f29275a0449eb18312be6b777e7c" is_pre_election: true
I20260927 22:51:55.546777 19160 catalog_manager.cc:5495] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 74b6f29275a0449eb18312be6b777e7c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "74b6f29275a0449eb18312be6b777e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b6f29275a0449eb18312be6b777e7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:55.548763 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.532700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40407 (local address 127.0.97.1:58645)
0927 22:51:55.541898 (+ 9198us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.541918 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.543491 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.544033 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.544037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.544068 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:55.544225 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.544234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.545542 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.545547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.546928 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.546940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.548088 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.548094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.548270 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.548294 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.548368 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.548495 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9067,"thread_start_us":11096,"threads_started":1}
I20260927 22:51:55.548784 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.533092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58645 (local address 127.0.97.2:40407)
0927 22:51:55.533249 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.533254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.541997 (+ 8743us) server_negotiation.cc:407] Connection header received
0927 22:51:55.543678 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.543684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.543760 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.543821 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:55.544363 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.545392 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.547075 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.547945 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.548057 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.548153 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.548407 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.548526 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:51:55.549400 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1d9cfa64dd454a9a93f5496a759a7e" candidate_uuid: "ec51f8251413404b958865166d7f2328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8b78092dd1453e8479df821eaf1025" is_pre_election: true
I20260927 22:51:55.549711 19432 raft_consensus.cc:2379] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec51f8251413404b958865166d7f2328 in current term 1: Already voted for candidate 74b6f29275a0449eb18312be6b777e7c in this term.
I20260927 22:51:55.550348 19234 leader_election.cc:302] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec51f8251413404b958865166d7f2328; no voters: 74b6f29275a0449eb18312be6b777e7c, bd8b78092dd1453e8479df821eaf1025
I20260927 22:51:55.550720 19633 raft_consensus.cc:2726] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:51:55.641623 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.636226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.0.97.3:44591)
0927 22:51:55.636395 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.636400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.636787 (+ 387us) server_negotiation.cc:407] Connection header received
0927 22:51:55.636962 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.636966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.637060 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.637178 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:55.637544 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.638272 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.639748 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.640132 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.640215 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.640674 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:55.641154 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:55.641275 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.641350 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.641400 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:51:55.648717 19432 raft_consensus.cc:1261] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 1 FOLLOWER]: Refusing update from remote peer 74b6f29275a0449eb18312be6b777e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:55.648800 19301 raft_consensus.cc:1261] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 1 FOLLOWER]: Refusing update from remote peer 74b6f29275a0449eb18312be6b777e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:55.649509 19637 consensus_queue.cc:1035] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8b78092dd1453e8479df821eaf1025" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:55.649807 19632 consensus_queue.cc:1035] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec51f8251413404b958865166d7f2328" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:55.661610 19643 mvcc.cc:205] Tried to move back new op lower bound from 7334090816088514560 to 7334090815662788608. Current Snapshot: MvccSnapshot[applied={T|T < 7334090816088514560}]
I20260927 22:51:55.673723 19159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:55.673991 19159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:55.677378 19159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5c98ed52d34466879b934273e292d5: Sending DeleteTablet for 3 replicas of tablet ba1d9cfa64dd454a9a93f5496a759a7e
I20260927 22:51:55.678468 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1d9cfa64dd454a9a93f5496a759a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:55 UTC) from {username='slave'} at 127.0.0.1:44984
I20260927 22:51:55.678474 19542 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1d9cfa64dd454a9a93f5496a759a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:55 UTC) from {username='slave'} at 127.0.0.1:37902
I20260927 22:51:55.678596 19412 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1d9cfa64dd454a9a93f5496a759a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:55 UTC) from {username='slave'} at 127.0.0.1:56344
I20260927 22:51:55.680436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 19219
I20260927 22:51:55.691710 19654 tablet_replica.cc:330] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: stopping tablet replica
I20260927 22:51:55.692353 19654 raft_consensus.cc:2229] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:55.693022 19654 raft_consensus.cc:2258] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:55.693984 19652 tablet_replica.cc:330] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: stopping tablet replica
I20260927 22:51:55.694635 19652 raft_consensus.cc:2229] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:55.695495 19654 ts_tablet_manager.cc:1905] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:55.695516 19652 raft_consensus.cc:2258] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:55.697675 19652 ts_tablet_manager.cc:1905] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:55.702488 19654 ts_tablet_manager.cc:1918] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:55.705658 19652 ts_tablet_manager.cc:1918] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:55.705919 19652 log.cc:1196] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/ba1d9cfa64dd454a9a93f5496a759a7e
I20260927 22:51:55.706519 19652 ts_tablet_manager.cc:1939] T ba1d9cfa64dd454a9a93f5496a759a7e P 74b6f29275a0449eb18312be6b777e7c: Deleting consensus metadata
I20260927 22:51:55.708602 19146 catalog_manager.cc:4841] TS 74b6f29275a0449eb18312be6b777e7c (127.0.97.3:44591): tablet ba1d9cfa64dd454a9a93f5496a759a7e (table test-table [id=92c370898ae645b6a766b2d3126833fc]) successfully deleted
I20260927 22:51:55.709093 19653 tablet_replica.cc:330] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: stopping tablet replica
I20260927 22:51:55.709539 19653 raft_consensus.cc:2229] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:55.709674 19654 log.cc:1196] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/ba1d9cfa64dd454a9a93f5496a759a7e
I20260927 22:51:55.710106 19653 raft_consensus.cc:2258] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:55.710216 19654 ts_tablet_manager.cc:1939] T ba1d9cfa64dd454a9a93f5496a759a7e P bd8b78092dd1453e8479df821eaf1025: Deleting consensus metadata
I20260927 22:51:55.711493 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.681831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.97.1:39619)
0927 22:51:55.682015 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.682021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.693943 (+ 11922us) server_negotiation.cc:407] Connection header received
0927 22:51:55.703014 (+ 9071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.703021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.703189 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.703276 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:55.703965 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.705146 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.707049 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.707730 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.707829 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.709658 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:55.709671 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:55.709688 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:55.710491 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:55.710552 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:55.710576 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:55.710683 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.710952 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.711060 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:51:55.712011 19144 catalog_manager.cc:4841] TS bd8b78092dd1453e8479df821eaf1025 (127.0.97.2:40407): tablet ba1d9cfa64dd454a9a93f5496a759a7e (table test-table [id=92c370898ae645b6a766b2d3126833fc]) successfully deleted
I20260927 22:51:55.712626 19341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:55.712878 19653 ts_tablet_manager.cc:1905] T ba1d9cfa64dd454a9a93f5496a759a7e P ec51f8251413404b958865166d7f2328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:55.820654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 19219
W20260927 22:51:55.835189 19147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39619 (error 108)
I20260927 22:51:55.835894 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 19350
I20260927 22:51:55.837744 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.836433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:45004)
0927 22:51:55.837335 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.837505 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":618,"threads_started":1}
W20260927 22:51:55.838064 19147 catalog_manager.cc:4584] TS ec51f8251413404b958865166d7f2328 (127.0.97.1:39619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba1d9cfa64dd454a9a93f5496a759a7e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
I20260927 22:51:55.844260 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.836676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.0.97.2:40407)
0927 22:51:55.836899 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.836906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.837657 (+ 751us) server_negotiation.cc:407] Connection header received
0927 22:51:55.837995 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.838000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.838114 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.838222 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:55.838722 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.839601 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.841453 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.841826 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.841942 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.843013 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:55.843021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:55.843034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:55.843633 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:55.843678 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:55.843703 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:55.843784 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.843926 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.843999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:51:55.844916 19472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:55.891762 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.891168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:45016)
0927 22:51:55.891455 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.891571 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 22:51:55.893011 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.892558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:45026)
0927 22:51:55.892743 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.892836 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:51:55.932186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 19350
I20260927 22:51:55.953172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 19480
I20260927 22:51:55.953888 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.953428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:45036)
0927 22:51:55.953618 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.953710 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:51:55.955320 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.954830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:45048)
0927 22:51:55.955006 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.955087 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:51:55.960611 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.954025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.0.97.3:44591)
0927 22:51:55.954365 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.954371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.954389 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:51:55.954446 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.954453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.954564 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.954648 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:55.955348 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.956349 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.957672 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.958095 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.958211 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.959231 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:55.959239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:55.959252 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:55.959933 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:55.959990 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:55.960010 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:55.960109 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.960239 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.960329 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260927 22:51:55.961284 19602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:56.035869 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.035363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:45064)
0927 22:51:56.035629 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.035723 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:51:56.036949 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.036520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39619 (local address 127.0.0.1:45072)
0927 22:51:56.036728 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.036812 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 22:51:56.079272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 19480
I20260927 22:51:56.094440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 19129
I20260927 22:51:56.095584 19190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:56.201236 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 19129
2026-09-27T22:51:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3123 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:56 dist-test-slave-kr25 krb5kdc[19664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:56 dist-test-slave-kr25 krb5kdc[19664](info): set up 2 sockets
Sep 27 22:51:56 dist-test-slave-kr25 krb5kdc[19664](info): commencing operation
krb5kdc: starting...
W20260927 22:51:57.211323 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:57 dist-test-slave-kr25 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:57.439886 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34637
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:57.753669 19677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:57.753947 19677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:57.754072 19677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:57.761116 19677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:57.761217 19677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:57.761286 19677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:57.761359 19677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:57.761412 19677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:57.770915 19677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:57.772236 19677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:57.773605 19677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:57.785130 19682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:57.785166 19685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:57.785203 19683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:57.786005 19677 server_base.cc:1034] running on GCE node
I20260927 22:51:57.786753 19677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:57.788161 19677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:57.789364 19677 hybrid_clock.cc:648] HybridClock initialized: now 1790549517789318 us; error 32 us; skew 500 ppm
I20260927 22:51:57.789695 19677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:57 dist-test-slave-kr25 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:57.796501 19677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:57.797938 19677 webserver.cc:460] Webserver started at http://127.0.97.62:44575/ using document root <none> and password file <none>
I20260927 22:51:57.798599 19677 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:57.798710 19677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:57.798997 19677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:57.801870 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "3a11e89143234c82bf6bbc741ea4dfd4"
format_stamp: "Formatted at 2026-09-27 22:51:57 on dist-test-slave-kr25"
server_key: "154e2a2ac1cecf7225c38d1f564dcd4a"
server_key_iv: "973f8956f017fa2acbe87fd91047a557"
server_key_version: "encryptionkey@0"
I20260927 22:51:57.802565 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "3a11e89143234c82bf6bbc741ea4dfd4"
format_stamp: "Formatted at 2026-09-27 22:51:57 on dist-test-slave-kr25"
server_key: "154e2a2ac1cecf7225c38d1f564dcd4a"
server_key_iv: "973f8956f017fa2acbe87fd91047a557"
server_key_version: "encryptionkey@0"
I20260927 22:51:57.807526 19677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:51:57.810832 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:57.812239 19677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:51:57.812412 19677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "3a11e89143234c82bf6bbc741ea4dfd4"
format_stamp: "Formatted at 2026-09-27 22:51:57 on dist-test-slave-kr25"
server_key: "154e2a2ac1cecf7225c38d1f564dcd4a"
server_key_iv: "973f8956f017fa2acbe87fd91047a557"
server_key_version: "encryptionkey@0"
I20260927 22:51:57.812621 19677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:57.827566 19677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:57.831673 19677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:57.831964 19677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:57.848574 19677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34637
I20260927 22:51:57.848605 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34637 every 8 connection(s)
I20260927 22:51:57.849869 19677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:57.850806 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 19677
I20260927 22:51:57.851024 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:57.851586 388 external_mini_cluster.cc:1428] Setting key 3f640000ebe4e5580fe9a7357c67e760
I20260927 22:51:57.854420 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:51:57 dist-test-slave-kr25 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:57.864868 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4: Bootstrap starting.
I20260927 22:51:57.867909 19744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:57.868904 19744 log.cc:824] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:57.871565 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4: No bootstrap required, opened a new log
I20260927 22:51:57.872545 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.853501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.0.97.62:34637)
0927 22:51:57.854262 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.854274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.854315 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:51:57.856051 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.856075 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.856473 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.856841 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.858336 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.859722 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.860993 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.861655 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.861837 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.865867 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.865875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.865889 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.865926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.868721 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.869386 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.869392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.869398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.869577 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.869914 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.869917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.869919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.870406 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.870691 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.871124 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.871565 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":342,"threads_started":1}
I20260927 22:51:57.878748 19744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260927 22:51:57.879084 19744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:57.879168 19744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a11e89143234c82bf6bbc741ea4dfd4, State: Initialized, Role: FOLLOWER
I20260927 22:51:57.879719 19744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260927 22:51:57.879940 19744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:57.880075 19744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:57.880228 19744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:57.882295 19744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260927 22:51:57.882755 19744 leader_election.cc:302] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a11e89143234c82bf6bbc741ea4dfd4; no voters:
I20260927 22:51:57.883257 19744 leader_election.cc:288] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:57.883380 19749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:57.884503 19749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a11e89143234c82bf6bbc741ea4dfd4, State: Running, Role: LEADER
I20260927 22:51:57.884995 19749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260927 22:51:57.885320 19744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:57.887954 19750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } } }
I20260927 22:51:57.888290 19750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:57.889456 19751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a11e89143234c82bf6bbc741ea4dfd4. Latest consensus state: current_term: 1 leader_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11e89143234c82bf6bbc741ea4dfd4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } } }
I20260927 22:51:57.889825 19751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:57.890317 19757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:57.895156 19757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:57.903894 19757 catalog_manager.cc:1347] Generated new cluster ID: 1bc5f21265ee4ef79731920a6df2e0f0
I20260927 22:51:57.904012 19757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:57.913511 19757 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:57.914816 19757 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:57.924926 19757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4: Generated new TSK 0
I20260927 22:51:57.925704 19757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:57.986840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:58.306089 19770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:58.306367 19770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:58.306497 19770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:58.313747 19770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:58.313858 19770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:58.314010 19770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:58.323791 19770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:58.325032 19770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:58.326294 19770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:58.338922 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.338894 19776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.339023 19775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:58.339913 19770 server_base.cc:1034] running on GCE node
I20260927 22:51:58.340497 19770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:58.341405 19770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:58.342605 19770 hybrid_clock.cc:648] HybridClock initialized: now 1790549518342554 us; error 44 us; skew 500 ppm
I20260927 22:51:58.342908 19770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:58 dist-test-slave-kr25 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:58.345885 19770 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:58.347294 19770 webserver.cc:460] Webserver started at http://127.0.97.1:41001/ using document root <none> and password file <none>
I20260927 22:51:58.347981 19770 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:58.348093 19770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:58.348368 19770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:58.351125 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "57ec0e2f0701491f8a1b273a283314bc"
format_stamp: "Formatted at 2026-09-27 22:51:58 on dist-test-slave-kr25"
server_key: "87b36f315810a6a1e3c570902a53cbe4"
server_key_iv: "af42471ebe4c1b5d8b6364b222936af0"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.351851 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "57ec0e2f0701491f8a1b273a283314bc"
format_stamp: "Formatted at 2026-09-27 22:51:58 on dist-test-slave-kr25"
server_key: "87b36f315810a6a1e3c570902a53cbe4"
server_key_iv: "af42471ebe4c1b5d8b6364b222936af0"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.356534 19770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:51:58.359779 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.361078 19770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:51:58.361215 19770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "57ec0e2f0701491f8a1b273a283314bc"
format_stamp: "Formatted at 2026-09-27 22:51:58 on dist-test-slave-kr25"
server_key: "87b36f315810a6a1e3c570902a53cbe4"
server_key_iv: "af42471ebe4c1b5d8b6364b222936af0"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.361392 19770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:58.388079 19770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:58.392498 19770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:58.392736 19770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:58.393595 19770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:58.394842 19770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:58.394912 19770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.395010 19770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:58.395052 19770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.424201 19770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38415
I20260927 22:51:58.424245 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38415 every 8 connection(s)
I20260927 22:51:58.425531 19770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:58.432375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 19770
I20260927 22:51:58.432529 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:58.432950 388 external_mini_cluster.cc:1428] Setting key ad99451b723a8c8bc9ef5aba0079e1ce
Sep 27 22:51:58 dist-test-slave-kr25 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:58.445542 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.427948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56107 (local address 127.0.97.62:34637)
0927 22:51:58.428315 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.428320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.429238 (+ 918us) server_negotiation.cc:407] Connection header received
0927 22:51:58.431023 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.431029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.431104 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.431194 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.432900 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.433728 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.435878 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.436268 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.436373 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.440191 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.440197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.440201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.440236 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.442750 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.443397 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.443402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.443406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.443558 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.443935 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.443939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.443942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.444237 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.444369 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.444980 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.445187 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":186,"threads_started":1}
I20260927 22:51:58.446295 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.428449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34637 (local address 127.0.97.1:56107)
0927 22:51:58.429100 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.429136 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.430770 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.431454 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.431466 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.431981 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.432732 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.432753 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.434390 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.434395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.435738 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.435750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.436478 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.436485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.436701 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.437363 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.437395 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.439977 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.442874 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.442880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.442893 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.443282 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.443644 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.443647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.443649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.443833 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.444336 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.444341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.444637 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.444851 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.445219 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":245,"threads_started":1}
I20260927 22:51:58.447731 19898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34637
I20260927 22:51:58.448016 19898 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.448685 19898 heartbeater.cc:511] Master 127.0.97.62:34637 requested a full tablet report, sending...
I20260927 22:51:58.450708 19709 ts_manager.cc:194] Registered new tserver with Master: 57ec0e2f0701491f8a1b273a283314bc (127.0.97.1:38415)
I20260927 22:51:58.452231 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56107
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:58.484732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:58.793071 19904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:58.793350 19904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:58.793494 19904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:58.800539 19904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:58.800648 19904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:58.800798 19904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:58.809949 19904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:58.811123 19904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:58.812541 19904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:58.825146 19910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.825109 19912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.825189 19909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:58.828785 19904 server_base.cc:1034] running on GCE node
I20260927 22:51:58.829362 19904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:58.830233 19904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:58.831470 19904 hybrid_clock.cc:648] HybridClock initialized: now 1790549518831401 us; error 32 us; skew 500 ppm
I20260927 22:51:58.831782 19904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:58 dist-test-slave-kr25 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:58.834685 19904 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:58.836066 19904 webserver.cc:460] Webserver started at http://127.0.97.2:41547/ using document root <none> and password file <none>
I20260927 22:51:58.836736 19904 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:58.836843 19904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:58.837121 19904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:58.839850 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "fd49ed94190642f29c09c526515edf97"
format_stamp: "Formatted at 2026-09-27 22:51:58 on dist-test-slave-kr25"
server_key: "4a2ba1790243c9cccc3769087d5436ae"
server_key_iv: "d1142228f95007dbd68ab8e15dd4ebe6"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.840579 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fd49ed94190642f29c09c526515edf97"
format_stamp: "Formatted at 2026-09-27 22:51:58 on dist-test-slave-kr25"
server_key: "4a2ba1790243c9cccc3769087d5436ae"
server_key_iv: "d1142228f95007dbd68ab8e15dd4ebe6"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.845196 19904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:51:58.848414 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.849707 19904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:58.849841 19904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "fd49ed94190642f29c09c526515edf97"
format_stamp: "Formatted at 2026-09-27 22:51:58 on dist-test-slave-kr25"
server_key: "4a2ba1790243c9cccc3769087d5436ae"
server_key_iv: "d1142228f95007dbd68ab8e15dd4ebe6"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.850003 19904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:58.861362 19904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:58.865523 19904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:58.865774 19904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:58.866640 19904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:58.867887 19904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:58.867964 19904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.868076 19904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:58.868127 19904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.896642 19904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38821
I20260927 22:51:58.896703 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38821 every 8 connection(s)
I20260927 22:51:58.897894 19904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:58.905037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 19904
I20260927 22:51:58.905225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:58.905781 388 external_mini_cluster.cc:1428] Setting key 60018b532869e3e6e61d4322577e1c84
Sep 27 22:51:58 dist-test-slave-kr25 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:58.917625 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.900380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44135 (local address 127.0.97.62:34637)
0927 22:51:58.900587 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.900594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.901726 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 22:51:58.903492 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.903496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.903562 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.903646 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.905076 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.906111 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.907672 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.908080 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.908191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.911652 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.911657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.911660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.911692 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.914058 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.914956 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.914962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.914966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.915203 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.915775 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.915781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.915784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.916128 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.916248 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.917087 (+ 839us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.917271 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:51:58.918725 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.900870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34637 (local address 127.0.97.2:44135)
0927 22:51:58.901566 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.901611 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.903225 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.903830 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.903837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.904352 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.904895 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.904920 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.906252 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.906256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.907536 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.907549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.908282 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.908289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.908504 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.909093 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.909114 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.911408 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.914183 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.914191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.914208 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.914780 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.915350 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.915356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.915360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.915654 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.916272 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.916280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.916689 (+ 409us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.916962 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.917452 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":265,"threads_started":1}
I20260927 22:51:58.920166 20032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34637
I20260927 22:51:58.920470 20032 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.921137 20032 heartbeater.cc:511] Master 127.0.97.62:34637 requested a full tablet report, sending...
I20260927 22:51:58.922484 19709 ts_manager.cc:194] Registered new tserver with Master: fd49ed94190642f29c09c526515edf97 (127.0.97.2:38821)
I20260927 22:51:58.923246 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44135
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:58.955716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:59.264166 20037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:59.264453 20037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:59.264590 20037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:59.271675 20037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:59.271836 20037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:59.271993 20037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:59.281358 20037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:51:59.282553 20037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:59.283815 20037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:59.296408 20043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:59.296451 20042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:59.296365 20045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:59.297154 20037 server_base.cc:1034] running on GCE node
I20260927 22:51:59.297737 20037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:59.298705 20037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:59.299881 20037 hybrid_clock.cc:648] HybridClock initialized: now 1790549519299811 us; error 64 us; skew 500 ppm
I20260927 22:51:59.300170 20037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:59 dist-test-slave-kr25 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:59.303180 20037 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:59.304601 20037 webserver.cc:460] Webserver started at http://127.0.97.3:42907/ using document root <none> and password file <none>
I20260927 22:51:59.305264 20037 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:59.305370 20037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:59.305648 20037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:59.308471 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "26386f97f6bf4848a0064da7faaeb585"
format_stamp: "Formatted at 2026-09-27 22:51:59 on dist-test-slave-kr25"
server_key: "2ead5ccb4334f2ba0646928ef93843e5"
server_key_iv: "5fe4f7d97fed8e129dcf054ff3884b9c"
server_key_version: "encryptionkey@0"
I20260927 22:51:59.309170 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "26386f97f6bf4848a0064da7faaeb585"
format_stamp: "Formatted at 2026-09-27 22:51:59 on dist-test-slave-kr25"
server_key: "2ead5ccb4334f2ba0646928ef93843e5"
server_key_iv: "5fe4f7d97fed8e129dcf054ff3884b9c"
server_key_version: "encryptionkey@0"
I20260927 22:51:59.313920 20037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 22:51:59.317250 20052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:59.318516 20037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 22:51:59.318652 20037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "26386f97f6bf4848a0064da7faaeb585"
format_stamp: "Formatted at 2026-09-27 22:51:59 on dist-test-slave-kr25"
server_key: "2ead5ccb4334f2ba0646928ef93843e5"
server_key_iv: "5fe4f7d97fed8e129dcf054ff3884b9c"
server_key_version: "encryptionkey@0"
I20260927 22:51:59.318823 20037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:59.343286 20037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:59.347533 20037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:59.347803 20037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:59.348719 20037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:59.349973 20037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:59.350042 20037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:59.350145 20037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:59.350191 20037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:59.380786 20037 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43111
I20260927 22:51:59.380839 20164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43111 every 8 connection(s)
I20260927 22:51:59.382015 20037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:59.386195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20037
I20260927 22:51:59.386358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:59.386817 388 external_mini_cluster.cc:1428] Setting key 048776e1691ed8902c6cb8a4d31269cf
Sep 27 22:51:59 dist-test-slave-kr25 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:59.402271 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.384547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38697 (local address 127.0.97.62:34637)
0927 22:51:59.384767 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.384773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.385894 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 22:51:59.387998 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.388002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.388084 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.388179 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.390078 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.391150 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.392958 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.393342 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.393438 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.396758 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.396764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.396767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.396802 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.399162 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.399922 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.399926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.399928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.400045 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.400411 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.400416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.400419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.400780 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.400934 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.401681 (+ 747us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.401937 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:51:59.403223 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.385039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34637 (local address 127.0.97.3:38697)
0927 22:51:59.385750 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.385788 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.387736 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.388401 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.388413 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.389098 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:59.389858 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.389885 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.391309 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.391315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.392805 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.392818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.393537 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.393542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.393746 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.394378 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:59.394401 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:59.396552 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:59.399284 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:59.399291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:59.399305 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:59.399807 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:59.400123 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:59.400126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:59.400128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:59.400308 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:59.400928 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:59.400937 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:59.401300 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.401559 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.402046 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":294,"threads_started":1}
I20260927 22:51:59.404467 20165 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34637
I20260927 22:51:59.404744 20165 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:59.405401 20165 heartbeater.cc:511] Master 127.0.97.62:34637 requested a full tablet report, sending...
I20260927 22:51:59.406685 19709 ts_manager.cc:194] Registered new tserver with Master: 26386f97f6bf4848a0064da7faaeb585 (127.0.97.3:43111)
I20260927 22:51:59.407521 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38697
I20260927 22:51:59.415061 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:59.431757 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.419177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.0.97.62:34637)
0927 22:51:59.419369 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.419375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.419794 (+ 419us) server_negotiation.cc:407] Connection header received
0927 22:51:59.419969 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.419973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.420040 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.420114 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.421527 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.422295 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.423698 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.424290 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.424417 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.426527 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.426533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.426538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.426576 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.429597 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.430210 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.430214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.430216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.430338 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.430607 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.430610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.430612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.430901 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.431071 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.431343 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.431537 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:51:59.435323 19709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:59.442059 19709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:59.455219 19898 heartbeater.cc:503] Master 127.0.97.62:34637 was elected leader, sending a full tablet report...
I20260927 22:51:59.465252 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.453677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38415 (local address 127.0.0.1:60036)
0927 22:51:59.456605 (+ 2928us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.456652 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.456903 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.457939 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.457947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.457995 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.458927 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.458945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.460684 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.460688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.462578 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.462593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.464122 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.464129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.464314 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.464362 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.464517 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.464701 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2642,"mutex_wait_us":442,"spinlock_wait_cycles":1280,"thread_start_us":3170,"threads_started":1}
I20260927 22:51:59.465950 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.454853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.97.3:43111)
0927 22:51:59.456151 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.456160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.457149 (+ 989us) server_negotiation.cc:407] Connection header received
0927 22:51:59.457328 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.457334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.457511 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.457640 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.458626 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.460247 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.463289 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.464633 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.464746 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.464970 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.465486 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.465650 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":970,"threads_started":1}
I20260927 22:51:59.466202 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.457217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38821 (local address 127.0.0.1:53546)
0927 22:51:59.458950 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.458964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.459076 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.459615 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.459619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.459640 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.459983 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.459992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.461679 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.461687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.463651 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.463662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.465507 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.465513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.465669 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.465699 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.465778 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.465908 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1610,"thread_start_us":1456,"threads_started":1}
I20260927 22:51:59.466233 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.453872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.0.97.1:38415)
0927 22:51:59.454959 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.454969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.456754 (+ 1785us) server_negotiation.cc:407] Connection header received
0927 22:51:59.457166 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.457174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.457460 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.457626 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.459121 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.460566 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.462737 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.463938 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.464061 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.464330 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.464564 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.464725 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":1130,"threads_started":1}
I20260927 22:51:59.466495 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.457518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.0.97.2:38821)
0927 22:51:59.458384 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.458393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.459059 (+ 666us) server_negotiation.cc:407] Connection header received
0927 22:51:59.459215 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.459220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.459399 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.459608 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.460116 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.461520 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.463782 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.465060 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.465175 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.465351 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.466047 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.466206 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":783,"threads_started":1}
I20260927 22:51:59.466921 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.454270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43111 (local address 127.0.0.1:48558)
0927 22:51:59.457029 (+ 2759us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.457045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.457156 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.457946 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.457951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.457997 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.458393 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.458407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.460384 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.460389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.463129 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.463143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.465116 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.465126 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.465288 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.465318 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.466458 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.466583 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"mutex_wait_us":47,"spinlock_wait_cycles":5632,"thread_start_us":1880,"threads_started":1}
I20260927 22:51:59.469803 19833 tablet_service.cc:1467] Processing CreateTablet for tablet 6f9a3078b16349e692c3829a6674e8aa (DEFAULT_TABLE table=test-table [id=7a86391ef15f41b3a3f11bdea029bdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:59.469959 20100 tablet_service.cc:1467] Processing CreateTablet for tablet 6f9a3078b16349e692c3829a6674e8aa (DEFAULT_TABLE table=test-table [id=7a86391ef15f41b3a3f11bdea029bdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:59.469969 19967 tablet_service.cc:1467] Processing CreateTablet for tablet 6f9a3078b16349e692c3829a6674e8aa (DEFAULT_TABLE table=test-table [id=7a86391ef15f41b3a3f11bdea029bdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:59.471248 19833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f9a3078b16349e692c3829a6674e8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:59.471302 20100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f9a3078b16349e692c3829a6674e8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:59.471303 19967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f9a3078b16349e692c3829a6674e8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:59.481945 20183 tablet_bootstrap.cc:492] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Bootstrap starting.
I20260927 22:51:59.485234 20184 tablet_bootstrap.cc:492] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Bootstrap starting.
I20260927 22:51:59.486205 20185 tablet_bootstrap.cc:492] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Bootstrap starting.
I20260927 22:51:59.486871 20183 tablet_bootstrap.cc:654] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:59.488260 20183 log.cc:824] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:59.488634 20184 tablet_bootstrap.cc:654] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:59.489220 20185 tablet_bootstrap.cc:654] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:59.489871 20184 log.cc:824] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:59.490195 20185 log.cc:824] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:59.492926 20184 tablet_bootstrap.cc:492] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: No bootstrap required, opened a new log
I20260927 22:51:59.493197 20184 ts_tablet_manager.cc:1397] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 22:51:59.493242 20185 tablet_bootstrap.cc:492] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: No bootstrap required, opened a new log
I20260927 22:51:59.493553 20185 ts_tablet_manager.cc:1397] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 22:51:59.494762 20183 tablet_bootstrap.cc:492] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: No bootstrap required, opened a new log
I20260927 22:51:59.495044 20183 ts_tablet_manager.cc:1397] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260927 22:51:59.500159 20184 raft_consensus.cc:348] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.500541 20184 raft_consensus.cc:374] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:59.500653 20184 raft_consensus.cc:729] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd49ed94190642f29c09c526515edf97, State: Initialized, Role: FOLLOWER
I20260927 22:51:59.501223 20184 consensus_queue.cc:260] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.502316 20032 heartbeater.cc:503] Master 127.0.97.62:34637 was elected leader, sending a full tablet report...
I20260927 22:51:59.502902 20184 ts_tablet_manager.cc:1428] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 22:51:59.503683 20185 raft_consensus.cc:348] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.504132 20185 raft_consensus.cc:374] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:59.504225 20185 raft_consensus.cc:729] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26386f97f6bf4848a0064da7faaeb585, State: Initialized, Role: FOLLOWER
I20260927 22:51:59.504787 20185 consensus_queue.cc:260] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.505774 20165 heartbeater.cc:503] Master 127.0.97.62:34637 was elected leader, sending a full tablet report...
I20260927 22:51:59.506453 20185 ts_tablet_manager.cc:1428] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260927 22:51:59.506598 20183 raft_consensus.cc:348] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.506943 20183 raft_consensus.cc:374] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:59.507074 20183 raft_consensus.cc:729] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ec0e2f0701491f8a1b273a283314bc, State: Initialized, Role: FOLLOWER
I20260927 22:51:59.507668 20183 consensus_queue.cc:260] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.509243 20183 ts_tablet_manager.cc:1428] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260927 22:51:59.586212 20190 raft_consensus.cc:482] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:59.586511 20190 raft_consensus.cc:504] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.586737 20189 raft_consensus.cc:482] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:59.587044 20189 raft_consensus.cc:504] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.588326 20190 leader_election.cc:288] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ec0e2f0701491f8a1b273a283314bc (127.0.97.1:38415), fd49ed94190642f29c09c526515edf97 (127.0.97.2:38821)
I20260927 22:51:59.588708 20189 leader_election.cc:288] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ec0e2f0701491f8a1b273a283314bc (127.0.97.1:38415), 26386f97f6bf4848a0064da7faaeb585 (127.0.97.3:43111)
I20260927 22:51:59.596582 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.588594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38415 (local address 127.0.97.3:47033)
0927 22:51:59.589180 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.589200 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.589341 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.590531 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.590536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.590560 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.590741 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.590770 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.592259 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.592265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.594326 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.594341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.595758 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.595764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.596017 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.596052 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.596149 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.596279 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":26}
I20260927 22:51:59.597450 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.588699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47033 (local address 127.0.97.1:38415)
0927 22:51:59.588899 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.588905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.590245 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 22:51:59.590332 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.590338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.590417 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.590515 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.590923 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.592069 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.594527 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.595612 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.595734 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.595833 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.597068 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.597200 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:59.598471 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9a3078b16349e692c3829a6674e8aa" candidate_uuid: "26386f97f6bf4848a0064da7faaeb585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ec0e2f0701491f8a1b273a283314bc" is_pre_election: true
I20260927 22:51:59.598866 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.589705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32897 (local address 127.0.97.2:38821)
0927 22:51:59.589910 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.589916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.590424 (+ 508us) server_negotiation.cc:407] Connection header received
0927 22:51:59.590605 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.590611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.590678 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.590742 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.591429 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.592495 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.596545 (+ 4050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.597781 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.597900 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.597986 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.598511 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.598643 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:51:59.599162 19853 raft_consensus.cc:2454] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26386f97f6bf4848a0064da7faaeb585 in term 0.
I20260927 22:51:59.599574 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.588940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38821 (local address 127.0.97.3:32897)
0927 22:51:59.590333 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.590347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.590466 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.590922 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.590926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.590951 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.591261 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.591273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.592677 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.592682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.596376 (+ 3694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.596389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.598150 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.598156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.598324 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.598356 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.599077 (+ 721us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.599212 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":1,"thread_start_us":1133,"threads_started":1}
I20260927 22:51:59.600001 20055 leader_election.cc:302] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26386f97f6bf4848a0064da7faaeb585, 57ec0e2f0701491f8a1b273a283314bc; no voters:
I20260927 22:51:59.600826 20190 raft_consensus.cc:2781] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:59.600980 20190 raft_consensus.cc:482] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:59.601073 20190 raft_consensus.cc:3037] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:59.601184 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9a3078b16349e692c3829a6674e8aa" candidate_uuid: "26386f97f6bf4848a0064da7faaeb585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd49ed94190642f29c09c526515edf97" is_pre_election: true
I20260927 22:51:59.601640 19987 raft_consensus.cc:2454] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26386f97f6bf4848a0064da7faaeb585 in term 0.
I20260927 22:51:59.604092 20190 raft_consensus.cc:504] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.604259 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.589974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47787 (local address 127.0.97.1:38415)
0927 22:51:59.597669 (+ 7695us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.597675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.597692 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:51:59.597749 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.597753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.597826 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.597899 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.598725 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.599886 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.602321 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.603169 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.603239 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.603304 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.603873 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.603998 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7601,"thread_start_us":10158,"threads_started":1}
I20260927 22:51:59.604954 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.588837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38415 (local address 127.0.97.2:47787)
0927 22:51:59.594734 (+ 5897us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.594751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.594864 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.598184 (+ 3320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.598189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.598218 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.598574 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.598586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.600058 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.600063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.602153 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.602166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.603489 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.603496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.603664 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.603717 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.604425 (+ 708us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.604565 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5754,"mutex_wait_us":2,"thread_start_us":11719,"threads_started":1}
I20260927 22:51:59.605012 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9a3078b16349e692c3829a6674e8aa" candidate_uuid: "26386f97f6bf4848a0064da7faaeb585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ec0e2f0701491f8a1b273a283314bc"
I20260927 22:51:59.605155 20190 leader_election.cc:288] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [CANDIDATE]: Term 1 election: Requested vote from peers 57ec0e2f0701491f8a1b273a283314bc (127.0.97.1:38415), fd49ed94190642f29c09c526515edf97 (127.0.97.2:38821)
I20260927 22:51:59.605237 19853 raft_consensus.cc:3037] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:59.605983 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9a3078b16349e692c3829a6674e8aa" candidate_uuid: "26386f97f6bf4848a0064da7faaeb585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd49ed94190642f29c09c526515edf97"
I20260927 22:51:59.606153 19987 raft_consensus.cc:3037] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:59.606108 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9a3078b16349e692c3829a6674e8aa" candidate_uuid: "fd49ed94190642f29c09c526515edf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ec0e2f0701491f8a1b273a283314bc" is_pre_election: true
I20260927 22:51:59.607803 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.589335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53935 (local address 127.0.97.3:43111)
0927 22:51:59.589682 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.589688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.600978 (+ 11290us) server_negotiation.cc:407] Connection header received
0927 22:51:59.601218 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.601224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.601290 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.601346 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.602383 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.603447 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.605661 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.606755 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.606845 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.606945 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.607481 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.607600 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260927 22:51:59.607985 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.589344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43111 (local address 127.0.97.2:53935)
0927 22:51:59.600870 (+ 11526us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.600897 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.601076 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.601511 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.601516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.601538 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.602216 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.602230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.603572 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.603577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.605492 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.605507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.607149 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.607156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.607317 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.607346 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.607453 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.607571 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11391,"mutex_wait_us":20,"thread_start_us":4937,"threads_started":1}
I20260927 22:51:59.608762 20120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f9a3078b16349e692c3829a6674e8aa" candidate_uuid: "fd49ed94190642f29c09c526515edf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26386f97f6bf4848a0064da7faaeb585" is_pre_election: true
I20260927 22:51:59.609165 20120 raft_consensus.cc:2379] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd49ed94190642f29c09c526515edf97 in current term 1: Already voted for candidate 26386f97f6bf4848a0064da7faaeb585 in this term.
I20260927 22:51:59.609270 19987 raft_consensus.cc:2454] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26386f97f6bf4848a0064da7faaeb585 in term 1.
I20260927 22:51:59.609270 19853 raft_consensus.cc:2454] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26386f97f6bf4848a0064da7faaeb585 in term 1.
I20260927 22:51:59.609608 20055 leader_election.cc:302] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26386f97f6bf4848a0064da7faaeb585, 57ec0e2f0701491f8a1b273a283314bc; no voters:
I20260927 22:51:59.609853 20190 raft_consensus.cc:2781] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:59.609867 19920 leader_election.cc:302] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd49ed94190642f29c09c526515edf97; no voters: 26386f97f6bf4848a0064da7faaeb585, 57ec0e2f0701491f8a1b273a283314bc
I20260927 22:51:59.610273 20189 raft_consensus.cc:2726] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:51:59.610325 20190 raft_consensus.cc:686] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 1 LEADER]: Becoming Leader. State: Replica: 26386f97f6bf4848a0064da7faaeb585, State: Running, Role: LEADER
I20260927 22:51:59.610782 20190 consensus_queue.cc:237] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260927 22:51:59.615231 19709 catalog_manager.cc:5495] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26386f97f6bf4848a0064da7faaeb585 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "26386f97f6bf4848a0064da7faaeb585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26386f97f6bf4848a0064da7faaeb585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } health_report { overall_health: HEALTHY } } }
W20260927 22:51:59.633512 20166 tablet.cc:2369] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:59.649858 20033 tablet.cc:2369] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:59.656261 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.650827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.97.3:43111)
0927 22:51:59.651002 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.651007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.651047 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:51:59.651279 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.651283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.651372 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.651525 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:59.651838 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.652596 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.654085 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.654484 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.654607 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.655084 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:59.655664 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:59.655797 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.655891 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.655983 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:59.664033 19987 raft_consensus.cc:1261] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 1 FOLLOWER]: Refusing update from remote peer 26386f97f6bf4848a0064da7faaeb585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:59.664038 19853 raft_consensus.cc:1261] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 1 FOLLOWER]: Refusing update from remote peer 26386f97f6bf4848a0064da7faaeb585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:59.664816 20196 consensus_queue.cc:1035] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ec0e2f0701491f8a1b273a283314bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:59.665158 20190 consensus_queue.cc:1035] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd49ed94190642f29c09c526515edf97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:59.673930 20199 mvcc.cc:205] Tried to move back new op lower bound from 7334090832534040576 to 7334090832329244672. Current Snapshot: MvccSnapshot[applied={T|T < 7334090832534040576}]
I20260927 22:51:59.676601 20200 mvcc.cc:205] Tried to move back new op lower bound from 7334090832534040576 to 7334090832329244672. Current Snapshot: MvccSnapshot[applied={T|T < 7334090832534040576}]
W20260927 22:51:59.677737 19899 tablet.cc:2369] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:59.690335 19709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:59.690606 19709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:59.693903 19709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a11e89143234c82bf6bbc741ea4dfd4: Sending DeleteTablet for 3 replicas of tablet 6f9a3078b16349e692c3829a6674e8aa
I20260927 22:51:59.695158 19833 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f9a3078b16349e692c3829a6674e8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60036
I20260927 22:51:59.695291 20100 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f9a3078b16349e692c3829a6674e8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48558
I20260927 22:51:59.695631 19967 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f9a3078b16349e692c3829a6674e8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53546
I20260927 22:51:59.696810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 19770
I20260927 22:51:59.707720 20209 tablet_replica.cc:330] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: stopping tablet replica
I20260927 22:51:59.708389 20209 raft_consensus.cc:2229] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:59.709069 20209 raft_consensus.cc:2258] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:59.709877 20210 tablet_replica.cc:330] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: stopping tablet replica
I20260927 22:51:59.710196 20210 raft_consensus.cc:2229] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:59.710767 20210 raft_consensus.cc:2258] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:59.712574 20209 ts_tablet_manager.cc:1905] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:59.717450 20210 ts_tablet_manager.cc:1905] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:59.721570 20211 tablet_replica.cc:330] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: stopping tablet replica
I20260927 22:51:59.721710 20209 ts_tablet_manager.cc:1918] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:59.721865 20209 log.cc:1196] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-0/wal/wals/6f9a3078b16349e692c3829a6674e8aa
I20260927 22:51:59.722059 20211 raft_consensus.cc:2229] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:59.722383 20209 ts_tablet_manager.cc:1939] T 6f9a3078b16349e692c3829a6674e8aa P 57ec0e2f0701491f8a1b273a283314bc: Deleting consensus metadata
I20260927 22:51:59.722580 20211 raft_consensus.cc:2258] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:59.724172 19695 catalog_manager.cc:4841] TS 57ec0e2f0701491f8a1b273a283314bc (127.0.97.1:38415): tablet 6f9a3078b16349e692c3829a6674e8aa (table test-table [id=7a86391ef15f41b3a3f11bdea029bdbf]) successfully deleted
I20260927 22:51:59.725014 20211 ts_tablet_manager.cc:1905] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:59.727638 20210 ts_tablet_manager.cc:1918] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:59.727867 20210 log.cc:1196] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/6f9a3078b16349e692c3829a6674e8aa
I20260927 22:51:59.728438 20210 ts_tablet_manager.cc:1939] T 6f9a3078b16349e692c3829a6674e8aa P 26386f97f6bf4848a0064da7faaeb585: Deleting consensus metadata
I20260927 22:51:59.730183 19693 catalog_manager.cc:4841] TS 26386f97f6bf4848a0064da7faaeb585 (127.0.97.3:43111): tablet 6f9a3078b16349e692c3829a6674e8aa (table test-table [id=7a86391ef15f41b3a3f11bdea029bdbf]) successfully deleted
Sep 27 22:51:59 dist-test-slave-kr25 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:59.731590 20211 ts_tablet_manager.cc:1918] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:59.731752 20211 log.cc:1196] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/6f9a3078b16349e692c3829a6674e8aa
I20260927 22:51:59.732261 20211 ts_tablet_manager.cc:1939] T 6f9a3078b16349e692c3829a6674e8aa P fd49ed94190642f29c09c526515edf97: Deleting consensus metadata
I20260927 22:51:59.734019 19695 catalog_manager.cc:4841] TS fd49ed94190642f29c09c526515edf97 (127.0.97.2:38821): tablet 6f9a3078b16349e692c3829a6674e8aa (table test-table [id=7a86391ef15f41b3a3f11bdea029bdbf]) successfully deleted
I20260927 22:51:59.739043 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.698111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.97.1:38415)
0927 22:51:59.698296 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.698302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.716034 (+ 17732us) server_negotiation.cc:407] Connection header received
0927 22:51:59.716319 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.716325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.716441 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.716519 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.718323 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.719340 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.721825 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.722493 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.722597 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.731534 (+ 8937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.731549 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.731564 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.731608 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.736218 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.736942 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.736952 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.736956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.737112 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.737471 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.737476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.737479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.737953 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.738273 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.738427 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.738687 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:51:59.739735 19893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:59.826846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 19770
I20260927 22:51:59.847402 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 19904
Sep 27 22:51:59 dist-test-slave-kr25 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:59.866688 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.848186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.0.97.2:38821)
0927 22:51:59.848369 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.848375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.848516 (+ 141us) server_negotiation.cc:407] Connection header received
0927 22:51:59.848701 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.848708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.848801 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.848924 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.850394 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.851490 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.853296 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.853952 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.854074 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.859594 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.859614 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.859635 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.859680 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.863238 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.864294 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.864305 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.864309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.864463 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.864869 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.864873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.864877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.865342 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.865625 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.865914 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.866170 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:51:59.867408 20027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:00.001829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 19904
I20260927 22:52:00.016294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20037
Sep 27 22:52:00 dist-test-slave-kr25 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:52:00.031785 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.017056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.97.3:43111)
0927 22:52:00.017361 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:00.017366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:00.017387 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:52:00.017556 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:00.017562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:00.017633 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:00.017718 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:00.019120 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.019907 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.021188 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.021623 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.021736 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:00.026021 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:00.026036 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:00.026052 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:00.026092 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:00.029073 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:00.029742 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:00.029749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:00.029751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:00.029873 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:00.030237 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:00.030240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:00.030243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:00.030632 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:00.030939 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:00.031212 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:52:00.031459 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 22:52:00.032532 20160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:00.151750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20037
I20260927 22:52:00.167201 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 19677
I20260927 22:52:00.168375 19739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:00.275581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 19677
2026-09-27T22:52:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4074 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T22:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:00Z Disabled control of system clock
I20260927 22:52:00.343043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33591
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:52:00.658533 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:00.658831 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:00.658962 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:00.667786 20220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:00.667929 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:00.668045 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:00.668134 20220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:00.668198 20220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:00.677961 20220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:00.679207 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:00.680943 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:00.691880 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:00.691911 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:00.692164 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:00.692962 20220 server_base.cc:1034] running on GCE node
I20260927 22:52:00.693681 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:00.695000 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:00.696199 20220 hybrid_clock.cc:648] HybridClock initialized: now 1790549520696152 us; error 40 us; skew 500 ppm
I20260927 22:52:00.696550 20220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:00.699023 20220 webserver.cc:460] Webserver started at http://127.0.97.62:41105/ using document root <none> and password file <none>
I20260927 22:52:00.699841 20220 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:00.699944 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:00.700237 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:00.703118 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "c95cb63a11324efe94146818ed75ea5c"
format_stamp: "Formatted at 2026-09-27 22:52:00 on dist-test-slave-kr25"
server_key: "d191873745fab7b2945ad43872c9e069"
server_key_iv: "fa7618190794870bf0b52efae73caf84"
server_key_version: "encryptionkey@0"
I20260927 22:52:00.703884 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "c95cb63a11324efe94146818ed75ea5c"
format_stamp: "Formatted at 2026-09-27 22:52:00 on dist-test-slave-kr25"
server_key: "d191873745fab7b2945ad43872c9e069"
server_key_iv: "fa7618190794870bf0b52efae73caf84"
server_key_version: "encryptionkey@0"
I20260927 22:52:00.708702 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 22:52:00.711820 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:00.713057 20220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:52:00.713212 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "c95cb63a11324efe94146818ed75ea5c"
format_stamp: "Formatted at 2026-09-27 22:52:00 on dist-test-slave-kr25"
server_key: "d191873745fab7b2945ad43872c9e069"
server_key_iv: "fa7618190794870bf0b52efae73caf84"
server_key_version: "encryptionkey@0"
I20260927 22:52:00.713392 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:00.746045 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:00.746899 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:00.747155 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:00.763376 20220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33591
I20260927 22:52:00.763406 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33591 every 8 connection(s)
I20260927 22:52:00.764638 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:00.768105 20286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:00.774031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20220
I20260927 22:52:00.774189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:00.774605 388 external_mini_cluster.cc:1428] Setting key fbbbad1d6fd09d98be70fe1258e3ca43
I20260927 22:52:00.778815 20286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c: Bootstrap starting.
I20260927 22:52:00.781924 20286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:00.782927 20286 log.cc:824] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:00.785717 20286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c: No bootstrap required, opened a new log
I20260927 22:52:00.787379 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.776493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.0.97.62:33591)
0927 22:52:00.777487 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:00.777499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:00.777541 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:52:00.779287 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:00.779309 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:00.779815 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:00.780169 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:00.780731 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.781952 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.783254 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.783984 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.784142 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:00.785219 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:00.785227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:00.785240 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:00.785745 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:00.785821 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:00.785845 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:00.785950 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:00.786174 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 22:52:00.786480 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":332,"threads_started":1}
I20260927 22:52:00.793514 20286 raft_consensus.cc:348] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95cb63a11324efe94146818ed75ea5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260927 22:52:00.793833 20286 raft_consensus.cc:374] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:00.793923 20286 raft_consensus.cc:729] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95cb63a11324efe94146818ed75ea5c, State: Initialized, Role: FOLLOWER
I20260927 22:52:00.794411 20286 consensus_queue.cc:260] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95cb63a11324efe94146818ed75ea5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260927 22:52:00.794655 20286 raft_consensus.cc:388] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:00.794782 20286 raft_consensus.cc:482] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:00.794941 20286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:00.797199 20286 raft_consensus.cc:504] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95cb63a11324efe94146818ed75ea5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260927 22:52:00.797624 20286 leader_election.cc:302] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c95cb63a11324efe94146818ed75ea5c; no voters:
I20260927 22:52:00.798185 20286 leader_election.cc:288] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:00.798300 20291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:00.798933 20291 raft_consensus.cc:686] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [term 1 LEADER]: Becoming Leader. State: Replica: c95cb63a11324efe94146818ed75ea5c, State: Running, Role: LEADER
I20260927 22:52:00.799392 20291 consensus_queue.cc:237] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95cb63a11324efe94146818ed75ea5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } }
I20260927 22:52:00.800171 20286 sys_catalog.cc:564] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:00.801891 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c95cb63a11324efe94146818ed75ea5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95cb63a11324efe94146818ed75ea5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } } }
I20260927 22:52:00.802341 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:00.802350 20293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c95cb63a11324efe94146818ed75ea5c. Latest consensus state: current_term: 1 leader_uuid: "c95cb63a11324efe94146818ed75ea5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95cb63a11324efe94146818ed75ea5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33591 } } }
I20260927 22:52:00.802981 20293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:00.804595 20299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:00.809775 20299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:00.818136 20299 catalog_manager.cc:1347] Generated new cluster ID: 79628613cc464b1b83c3765ddf7bfb12
I20260927 22:52:00.818257 20299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:00.855195 20299 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:00.857034 20299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:00.867240 20299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c: Generated new TSK 0
I20260927 22:52:00.868090 20299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:00.883047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--builtin_ntp_servers=127.0.97.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:52:01.196193 20310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:01.196466 20310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:01.196596 20310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:01.203812 20310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:01.203908 20310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:01.204044 20310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:52:01.213289 20310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:01.214670 20310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:01.216539 20310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:01.231802 20315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:01.231839 20316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:01.231865 20318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:01.232527 20310 server_base.cc:1034] running on GCE node
I20260927 22:52:01.233131 20310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:01.234118 20310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:01.235349 20310 hybrid_clock.cc:648] HybridClock initialized: now 1790549521235292 us; error 47 us; skew 500 ppm
I20260927 22:52:01.235697 20310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:01.237883 20310 webserver.cc:460] Webserver started at http://127.0.97.1:41619/ using document root <none> and password file <none>
I20260927 22:52:01.238585 20310 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:01.238700 20310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:01.239010 20310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:01.241808 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data/instance:
uuid: "e5dce0ce393142b9a7ea9ec2cdae9436"
format_stamp: "Formatted at 2026-09-27 22:52:01 on dist-test-slave-kr25"
server_key: "1a8b8417844d33e6181a1d4a2b286c97"
server_key_iv: "745c54e8c764ab97b8a6f6e15e82922b"
server_key_version: "encryptionkey@0"
I20260927 22:52:01.242534 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e5dce0ce393142b9a7ea9ec2cdae9436"
format_stamp: "Formatted at 2026-09-27 22:52:01 on dist-test-slave-kr25"
server_key: "1a8b8417844d33e6181a1d4a2b286c97"
server_key_iv: "745c54e8c764ab97b8a6f6e15e82922b"
server_key_version: "encryptionkey@0"
I20260927 22:52:01.247298 20310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:52:01.250483 20324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:01.251781 20310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:01.251919 20310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/wal
uuid: "e5dce0ce393142b9a7ea9ec2cdae9436"
format_stamp: "Formatted at 2026-09-27 22:52:01 on dist-test-slave-kr25"
server_key: "1a8b8417844d33e6181a1d4a2b286c97"
server_key_iv: "745c54e8c764ab97b8a6f6e15e82922b"
server_key_version: "encryptionkey@0"
I20260927 22:52:01.252122 20310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:01.279840 20310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:01.280694 20310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:01.280961 20310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:01.281877 20310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:01.283295 20310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:01.283385 20310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:01.283511 20310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:01.283581 20310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:01.313292 20310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43923
I20260927 22:52:01.313350 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43923 every 8 connection(s)
I20260927 22:52:01.314551 20310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:01.315336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20310
I20260927 22:52:01.315536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:01.316058 388 external_mini_cluster.cc:1428] Setting key 30a1ae3dae6719cc32303760010246bd
I20260927 22:52:01.319063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--builtin_ntp_servers=127.0.97.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:52:01.328622 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:01.317258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60277 (local address 127.0.97.62:33591)
0927 22:52:01.317658 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:01.317662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:01.319253 (+ 1591us) server_negotiation.cc:407] Connection header received
0927 22:52:01.321687 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:01.321692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:01.321759 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:01.321850 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:01.323133 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.323906 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.325915 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.326291 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.326376 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:01.327814 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:01.327821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:01.327826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:01.327849 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:01.327905 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:01.327920 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:01.328011 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:01.328195 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 22:52:01.328294 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":214,"threads_started":1}
I20260927 22:52:01.329511 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:01.317990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33591 (local address 127.0.97.1:60277)
0927 22:52:01.319104 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:01.319150 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:01.321394 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:01.322095 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:01.322105 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:01.322216 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:01.322963 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:01.322990 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.324048 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.324053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:01.325788 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:01.325803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.326485 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.326492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:01.326763 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:01.327483 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:01.327486 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:01.327546 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:01.327642 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:01.328020 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:01.328027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:01.328034 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:52:01.328169 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:52:01.328453 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":317,"threads_started":1}
I20260927 22:52:01.330997 20437 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33591
I20260927 22:52:01.331293 20437 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:01.332129 20437 heartbeater.cc:511] Master 127.0.97.62:33591 requested a full tablet report, sending...
I20260927 22:52:01.334332 20251 ts_manager.cc:194] Registered new tserver with Master: e5dce0ce393142b9a7ea9ec2cdae9436 (127.0.97.1:43923)
I20260927 22:52:01.336336 20251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:60277
W20260927 22:52:01.639882 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:01.640138 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:01.640229 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:01.647306 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:01.647408 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:01.647554 20441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:52:01.656657 20441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:01.657938 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:01.659770 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:01.672343 20446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:01.672369 20447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:01.672391 20449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:01.673103 20441 server_base.cc:1034] running on GCE node
I20260927 22:52:01.673727 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:01.674636 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:01.675829 20441 hybrid_clock.cc:648] HybridClock initialized: now 1790549521675752 us; error 70 us; skew 500 ppm
I20260927 22:52:01.676131 20441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:01.678359 20441 webserver.cc:460] Webserver started at http://127.0.97.2:38189/ using document root <none> and password file <none>
I20260927 22:52:01.679014 20441 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:01.679114 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:01.679420 20441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:01.682242 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data/instance:
uuid: "e9161eadb21949609dc406a60552665b"
format_stamp: "Formatted at 2026-09-27 22:52:01 on dist-test-slave-kr25"
server_key: "47275344e8b56fddc3a9dbf142d6bbb9"
server_key_iv: "7b34976001ed1c5fa3365a2e0929d272"
server_key_version: "encryptionkey@0"
I20260927 22:52:01.682997 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e9161eadb21949609dc406a60552665b"
format_stamp: "Formatted at 2026-09-27 22:52:01 on dist-test-slave-kr25"
server_key: "47275344e8b56fddc3a9dbf142d6bbb9"
server_key_iv: "7b34976001ed1c5fa3365a2e0929d272"
server_key_version: "encryptionkey@0"
I20260927 22:52:01.687937 20441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:52:01.691243 20455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:01.692617 20441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:01.692752 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal
uuid: "e9161eadb21949609dc406a60552665b"
format_stamp: "Formatted at 2026-09-27 22:52:01 on dist-test-slave-kr25"
server_key: "47275344e8b56fddc3a9dbf142d6bbb9"
server_key_iv: "7b34976001ed1c5fa3365a2e0929d272"
server_key_version: "encryptionkey@0"
I20260927 22:52:01.692940 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:01.731812 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:01.732618 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:01.732842 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:01.733729 20441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:01.735139 20441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:01.735220 20441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:01.735332 20441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:01.735394 20441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:01.763995 20441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34663
I20260927 22:52:01.764089 20567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34663 every 8 connection(s)
I20260927 22:52:01.765220 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:01.766762 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20441
I20260927 22:52:01.766921 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:01.767413 388 external_mini_cluster.cc:1428] Setting key 6d0d796ec29f45f7e983f1db68fc9193
I20260927 22:52:01.770371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--builtin_ntp_servers=127.0.97.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:52:01.780215 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:01.767819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41831 (local address 127.0.97.62:33591)
0927 22:52:01.768016 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:01.768022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:01.769886 (+ 1864us) server_negotiation.cc:407] Connection header received
0927 22:52:01.772318 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:01.772323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:01.772404 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:01.772499 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:01.773705 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.774697 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.777046 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.777670 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.777802 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:01.779404 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:01.779411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:01.779416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:01.779464 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:01.779502 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:01.779516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:01.779598 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:01.779811 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 22:52:01.779922 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:52:01.781591 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:01.768496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33591 (local address 127.0.97.2:41831)
0927 22:52:01.769731 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:01.769779 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:01.772035 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:01.772712 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:01.772723 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:01.772834 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:01.773538 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:01.773561 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.774856 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.774862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:01.776849 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:01.776866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:01.777912 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:01.777919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:01.778225 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:01.779033 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:01.779037 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:01.779089 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:01.779202 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:01.779600 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:01.779609 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:01.779617 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:52:01.779787 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:52:01.780206 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":309,"threads_started":1}
I20260927 22:52:01.782872 20568 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33591
I20260927 22:52:01.783190 20568 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:01.784073 20568 heartbeater.cc:511] Master 127.0.97.62:33591 requested a full tablet report, sending...
I20260927 22:52:01.785554 20251 ts_manager.cc:194] Registered new tserver with Master: e9161eadb21949609dc406a60552665b (127.0.97.2:34663)
I20260927 22:52:01.786314 20251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41831
W20260927 22:52:02.083033 20571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:02.083294 20571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:02.083405 20571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:02.090659 20571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:02.090750 20571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:02.090902 20571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:52:02.100000 20571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:02.101269 20571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:02.103012 20571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:02.115363 20576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:02.115329 20579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:02.115401 20577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:02.116079 20571 server_base.cc:1034] running on GCE node
I20260927 22:52:02.116707 20571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:02.117596 20571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:02.118788 20571 hybrid_clock.cc:648] HybridClock initialized: now 1790549522118738 us; error 44 us; skew 500 ppm
I20260927 22:52:02.119086 20571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:02.121114 20571 webserver.cc:460] Webserver started at http://127.0.97.3:41161/ using document root <none> and password file <none>
I20260927 22:52:02.121744 20571 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:02.121838 20571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:02.122126 20571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:02.124833 20571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data/instance:
uuid: "4d57e1ab82274e0294a6c12e421a08cd"
format_stamp: "Formatted at 2026-09-27 22:52:02 on dist-test-slave-kr25"
server_key: "8fb26d3f0d3fbb4ba6ad9d06d7e05e36"
server_key_iv: "4cdfe71278b58213145d7944d959cc17"
server_key_version: "encryptionkey@0"
I20260927 22:52:02.125554 20571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4d57e1ab82274e0294a6c12e421a08cd"
format_stamp: "Formatted at 2026-09-27 22:52:02 on dist-test-slave-kr25"
server_key: "8fb26d3f0d3fbb4ba6ad9d06d7e05e36"
server_key_iv: "4cdfe71278b58213145d7944d959cc17"
server_key_version: "encryptionkey@0"
I20260927 22:52:02.130162 20571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:52:02.133358 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:02.134693 20571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:52:02.134838 20571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal
uuid: "4d57e1ab82274e0294a6c12e421a08cd"
format_stamp: "Formatted at 2026-09-27 22:52:02 on dist-test-slave-kr25"
server_key: "8fb26d3f0d3fbb4ba6ad9d06d7e05e36"
server_key_iv: "4cdfe71278b58213145d7944d959cc17"
server_key_version: "encryptionkey@0"
I20260927 22:52:02.135030 20571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:02.189216 20571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:02.190056 20571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:02.190306 20571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:02.191375 20571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:02.192843 20571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:02.192938 20571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:02.193041 20571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:02.193109 20571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:02.222194 20571 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37297
I20260927 22:52:02.222249 20697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37297 every 8 connection(s)
I20260927 22:52:02.223412 20571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:02.225201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20571
I20260927 22:52:02.225383 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:02.225924 388 external_mini_cluster.cc:1428] Setting key a5984715271591618c87b72cfdca741c
I20260927 22:52:02.235263 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.226049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60179 (local address 127.0.97.62:33591)
0927 22:52:02.226216 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.226220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.228056 (+ 1836us) server_negotiation.cc:407] Connection header received
0927 22:52:02.229707 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.229711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.229775 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.229859 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:02.230880 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.231648 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.233085 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.233467 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.233551 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.234667 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:02.234671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:02.234675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:02.234692 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:02.234723 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:02.234734 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:02.234798 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.234970 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.235042 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:52:02.236186 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.226679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33591 (local address 127.0.97.3:60179)
0927 22:52:02.227901 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.227951 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.229483 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.230018 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.230026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.230149 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:02.230706 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.230722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.231762 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.231765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.232976 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.232987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.233616 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.233620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.233807 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.234375 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:02.234377 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:02.234413 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:02.234492 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:02.234801 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:02.234807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:02.234812 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.234955 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.235224 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":488,"threads_started":1}
I20260927 22:52:02.237353 20698 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33591
I20260927 22:52:02.237613 20698 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:02.238314 20698 heartbeater.cc:511] Master 127.0.97.62:33591 requested a full tablet report, sending...
I20260927 22:52:02.239748 20250 ts_manager.cc:194] Registered new tserver with Master: 4d57e1ab82274e0294a6c12e421a08cd (127.0.97.3:37297)
I20260927 22:52:02.240554 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:52:02.240845 20250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60179
I20260927 22:52:02.249233 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.244435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.97.62:33591)
0927 22:52:02.244590 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.244594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.244982 (+ 388us) server_negotiation.cc:407] Connection header received
0927 22:52:02.245116 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.245119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.245181 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.245248 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:02.245614 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.246253 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.247466 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.247858 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.247953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.248748 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:02.248752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:02.248755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:02.248771 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:02.248800 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:02.248811 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:02.248873 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.248975 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.249037 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:52:02.252413 20250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:52:02.258882 20250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:02.292949 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.270360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34663 (local address 127.0.0.1:41070)
0927 22:52:02.284501 (+ 14141us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.284514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.284618 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.285502 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.285508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.285537 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.285964 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.285978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.287681 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.287686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.290000 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.290017 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.292005 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.292011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.292210 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.292259 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.292386 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.292563 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14025,"mutex_wait_us":27,"thread_start_us":13907,"threads_started":1}
I20260927 22:52:02.292961 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.270785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.97.2:34663)
0927 22:52:02.282894 (+ 12109us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.282904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.284611 (+ 1707us) server_negotiation.cc:407] Connection header received
0927 22:52:02.284798 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.284806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.285067 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.285226 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.286176 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.287507 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.290187 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.291617 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.291734 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.291970 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.292458 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.292645 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11928,"thread_start_us":13126,"threads_started":1}
I20260927 22:52:02.294027 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.270066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35248)
0927 22:52:02.283538 (+ 13472us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.283582 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.283772 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.287164 (+ 3392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.287169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.287192 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.287589 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.287605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.288942 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.288947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.290904 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.290917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.292790 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.292796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.292966 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.292996 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.293579 (+ 583us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.293712 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13141,"mutex_wait_us":110,"thread_start_us":12989,"threads_started":1}
I20260927 22:52:02.294144 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.270088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.0.97.1:43923)
0927 22:52:02.286659 (+ 16571us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.286668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.286697 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:52:02.286784 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.286791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.286987 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.287168 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.287759 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.288811 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.291146 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.292336 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.292455 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.292662 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.293642 (+ 980us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.293802 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16440,"thread_start_us":16339,"threads_started":1}
I20260927 22:52:02.294314 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.284294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.0.97.3:37297)
0927 22:52:02.285070 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.285078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.286515 (+ 1437us) server_negotiation.cc:407] Connection header received
0927 22:52:02.286771 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.286778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.286986 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.287168 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.288126 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.289630 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.291812 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.293145 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.293238 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.293481 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.293835 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.293988 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":402,"threads_started":1}
I20260927 22:52:02.294916 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.285048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37297 (local address 127.0.0.1:57998)
0927 22:52:02.286417 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.286433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.286608 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.287429 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.287460 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.287490 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.287919 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.287931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.289794 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.289799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.291617 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.291630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.293486 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.293492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.293648 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.293679 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.294487 (+ 808us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.294619 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":22,"thread_start_us":1123,"threads_started":1}
I20260927 22:52:02.296823 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 37b112f1a79b4e2a97fa3ee298f10b94 (DEFAULT_TABLE table=test-table [id=7f3f414b776543a2a5fdba78768c861b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:02.298063 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b112f1a79b4e2a97fa3ee298f10b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:02.297852 20372 tablet_service.cc:1467] Processing CreateTablet for tablet 37b112f1a79b4e2a97fa3ee298f10b94 (DEFAULT_TABLE table=test-table [id=7f3f414b776543a2a5fdba78768c861b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:02.298359 20633 tablet_service.cc:1467] Processing CreateTablet for tablet 37b112f1a79b4e2a97fa3ee298f10b94 (DEFAULT_TABLE table=test-table [id=7f3f414b776543a2a5fdba78768c861b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:02.299256 20372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b112f1a79b4e2a97fa3ee298f10b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:02.299731 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b112f1a79b4e2a97fa3ee298f10b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:02.309091 20716 tablet_bootstrap.cc:492] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Bootstrap starting.
I20260927 22:52:02.312103 20716 tablet_bootstrap.cc:654] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:02.313228 20716 log.cc:824] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:02.319919 20718 tablet_bootstrap.cc:492] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Bootstrap starting.
I20260927 22:52:02.323704 20718 tablet_bootstrap.cc:654] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:02.324582 20717 tablet_bootstrap.cc:492] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: Bootstrap starting.
I20260927 22:52:02.325495 20718 log.cc:824] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:02.328301 20717 tablet_bootstrap.cc:654] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:02.328358 20718 tablet_bootstrap.cc:492] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: No bootstrap required, opened a new log
I20260927 22:52:02.328590 20718 ts_tablet_manager.cc:1397] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 22:52:02.329545 20717 log.cc:824] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:02.331679 20716 tablet_bootstrap.cc:492] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: No bootstrap required, opened a new log
I20260927 22:52:02.331934 20716 ts_tablet_manager.cc:1397] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.015s
I20260927 22:52:02.332242 20717 tablet_bootstrap.cc:492] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: No bootstrap required, opened a new log
I20260927 22:52:02.332502 20717 ts_tablet_manager.cc:1397] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:52:02.338614 20716 raft_consensus.cc:348] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.338560 20718 raft_consensus.cc:348] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.338989 20716 raft_consensus.cc:374] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:02.339006 20718 raft_consensus.cc:374] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:02.339085 20716 raft_consensus.cc:729] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9161eadb21949609dc406a60552665b, State: Initialized, Role: FOLLOWER
I20260927 22:52:02.339090 20718 raft_consensus.cc:729] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d57e1ab82274e0294a6c12e421a08cd, State: Initialized, Role: FOLLOWER
I20260927 22:52:02.339660 20716 consensus_queue.cc:260] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.339773 20718 consensus_queue.cc:260] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.340257 20437 heartbeater.cc:503] Master 127.0.97.62:33591 was elected leader, sending a full tablet report...
I20260927 22:52:02.342797 20717 raft_consensus.cc:348] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.343256 20717 raft_consensus.cc:374] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:02.343402 20717 raft_consensus.cc:729] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5dce0ce393142b9a7ea9ec2cdae9436, State: Initialized, Role: FOLLOWER
I20260927 22:52:02.344075 20717 consensus_queue.cc:260] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.346912 20698 heartbeater.cc:503] Master 127.0.97.62:33591 was elected leader, sending a full tablet report...
I20260927 22:52:02.347846 20568 heartbeater.cc:503] Master 127.0.97.62:33591 was elected leader, sending a full tablet report...
I20260927 22:52:02.349339 20718 ts_tablet_manager.cc:1428] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20260927 22:52:02.349831 20716 ts_tablet_manager.cc:1428] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Time spent starting tablet: real 0.018s user 0.007s sys 0.009s
I20260927 22:52:02.351246 20717 ts_tablet_manager.cc:1428] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260927 22:52:02.360467 20724 raft_consensus.cc:482] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:02.360699 20724 raft_consensus.cc:504] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.362250 20724 leader_election.cc:288] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d57e1ab82274e0294a6c12e421a08cd (127.0.97.3:37297), e9161eadb21949609dc406a60552665b (127.0.97.2:34663)
I20260927 22:52:02.368431 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.362617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33687 (local address 127.0.97.3:37297)
0927 22:52:02.362819 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.362825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.363246 (+ 421us) server_negotiation.cc:407] Connection header received
0927 22:52:02.363465 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.363471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.363532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.363605 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.364115 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.364813 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.366795 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.367620 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.367729 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.367819 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.368082 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.368195 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:52:02.368465 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.362430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37297 (local address 127.0.97.1:33687)
0927 22:52:02.363146 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.363162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.363300 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.363635 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.363639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.363663 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.363990 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.363999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.364936 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.364940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.366581 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.366595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.367780 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.367786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.367947 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.367971 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.368048 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.368171 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":424,"threads_started":1}
I20260927 22:52:02.368624 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.362648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58721 (local address 127.0.97.2:34663)
0927 22:52:02.362973 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.362980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.363566 (+ 586us) server_negotiation.cc:407] Connection header received
0927 22:52:02.363729 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.363732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.363798 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.363868 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.364358 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.365034 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.367014 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.367796 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.367908 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.368006 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.368279 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.368404 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260927 22:52:02.368857 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.362524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34663 (local address 127.0.97.1:58721)
0927 22:52:02.363470 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.363487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.363605 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.363904 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.363908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.363936 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.364222 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.364230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.365163 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.365168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.366845 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.366859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.367971 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.367977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.368135 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.368158 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.368231 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.368345 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":1,"thread_start_us":621,"threads_started":1}
I20260927 22:52:02.369798 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b112f1a79b4e2a97fa3ee298f10b94" candidate_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9161eadb21949609dc406a60552665b" is_pre_election: true
I20260927 22:52:02.369798 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b112f1a79b4e2a97fa3ee298f10b94" candidate_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d57e1ab82274e0294a6c12e421a08cd" is_pre_election: true
I20260927 22:52:02.370340 20523 raft_consensus.cc:2454] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5dce0ce393142b9a7ea9ec2cdae9436 in term 0.
I20260927 22:52:02.370337 20653 raft_consensus.cc:2454] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5dce0ce393142b9a7ea9ec2cdae9436 in term 0.
I20260927 22:52:02.370946 20325 leader_election.cc:302] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5dce0ce393142b9a7ea9ec2cdae9436, e9161eadb21949609dc406a60552665b; no voters:
I20260927 22:52:02.371287 20724 raft_consensus.cc:2781] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:02.371420 20724 raft_consensus.cc:482] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:02.371554 20724 raft_consensus.cc:3037] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:02.373658 20724 raft_consensus.cc:504] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.374275 20724 leader_election.cc:288] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [CANDIDATE]: Term 1 election: Requested vote from peers 4d57e1ab82274e0294a6c12e421a08cd (127.0.97.3:37297), e9161eadb21949609dc406a60552665b (127.0.97.2:34663)
I20260927 22:52:02.374667 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b112f1a79b4e2a97fa3ee298f10b94" candidate_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d57e1ab82274e0294a6c12e421a08cd"
I20260927 22:52:02.374756 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b112f1a79b4e2a97fa3ee298f10b94" candidate_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9161eadb21949609dc406a60552665b"
I20260927 22:52:02.374845 20653 raft_consensus.cc:3037] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:02.374954 20523 raft_consensus.cc:3037] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:02.377058 20653 raft_consensus.cc:2454] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5dce0ce393142b9a7ea9ec2cdae9436 in term 1.
I20260927 22:52:02.377166 20523 raft_consensus.cc:2454] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5dce0ce393142b9a7ea9ec2cdae9436 in term 1.
I20260927 22:52:02.377492 20326 leader_election.cc:302] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d57e1ab82274e0294a6c12e421a08cd, e5dce0ce393142b9a7ea9ec2cdae9436; no voters:
I20260927 22:52:02.377774 20724 raft_consensus.cc:2781] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:02.378202 20724 raft_consensus.cc:686] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 1 LEADER]: Becoming Leader. State: Replica: e5dce0ce393142b9a7ea9ec2cdae9436, State: Running, Role: LEADER
I20260927 22:52:02.378660 20724 consensus_queue.cc:237] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } }
I20260927 22:52:02.383253 20249 catalog_manager.cc:5495] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5dce0ce393142b9a7ea9ec2cdae9436 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dce0ce393142b9a7ea9ec2cdae9436" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 } health_report { overall_health: UNKNOWN } } }
I20260927 22:52:02.402021 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.396258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.0.97.1:43923)
0927 22:52:02.396439 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.396443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.396460 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:52:02.396604 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.396608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.396697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.396817 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:02.397140 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.397947 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.399376 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.400012 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.400165 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.400724 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:02.401444 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:02.401579 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.401667 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.401746 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:52:02.410024 20523 raft_consensus.cc:1261] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 1 FOLLOWER]: Refusing update from remote peer e5dce0ce393142b9a7ea9ec2cdae9436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:02.410046 20653 raft_consensus.cc:1261] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 1 FOLLOWER]: Refusing update from remote peer e5dce0ce393142b9a7ea9ec2cdae9436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:02.410898 20727 consensus_queue.cc:1035] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9161eadb21949609dc406a60552665b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:02.411208 20724 consensus_queue.cc:1035] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d57e1ab82274e0294a6c12e421a08cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:02.434789 20249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:02.435056 20249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:02.438169 20249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c95cb63a11324efe94146818ed75ea5c: Sending DeleteTablet for 3 replicas of tablet 37b112f1a79b4e2a97fa3ee298f10b94
I20260927 22:52:02.439203 20372 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b112f1a79b4e2a97fa3ee298f10b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:02 UTC) from {username='slave'} at 127.0.0.1:35248
I20260927 22:52:02.439321 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b112f1a79b4e2a97fa3ee298f10b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:02 UTC) from {username='slave'} at 127.0.0.1:41070
I20260927 22:52:02.440214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20310
I20260927 22:52:02.440219 20633 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b112f1a79b4e2a97fa3ee298f10b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:02 UTC) from {username='slave'} at 127.0.0.1:57998
I20260927 22:52:02.451861 20740 tablet_replica.cc:330] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: stopping tablet replica
I20260927 22:52:02.452342 20740 raft_consensus.cc:2229] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:02.452900 20740 raft_consensus.cc:2258] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:02.453025 20742 tablet_replica.cc:330] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: stopping tablet replica
I20260927 22:52:02.453414 20742 raft_consensus.cc:2229] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:02.453871 20742 raft_consensus.cc:2258] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:02.455140 20740 ts_tablet_manager.cc:1905] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:02.455862 20742 ts_tablet_manager.cc:1905] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:02.462368 20740 ts_tablet_manager.cc:1918] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:02.462514 20742 ts_tablet_manager.cc:1918] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:02.462668 20742 log.cc:1196] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-2/wal/wals/37b112f1a79b4e2a97fa3ee298f10b94
I20260927 22:52:02.462689 20740 log.cc:1196] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Deleting WAL directory at /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549216186161-388-0/minicluster-data/ts-1/wal/wals/37b112f1a79b4e2a97fa3ee298f10b94
I20260927 22:52:02.463160 20742 ts_tablet_manager.cc:1939] T 37b112f1a79b4e2a97fa3ee298f10b94 P 4d57e1ab82274e0294a6c12e421a08cd: Deleting consensus metadata
I20260927 22:52:02.463486 20740 ts_tablet_manager.cc:1939] T 37b112f1a79b4e2a97fa3ee298f10b94 P e9161eadb21949609dc406a60552665b: Deleting consensus metadata
I20260927 22:52:02.464970 20236 catalog_manager.cc:4841] TS 4d57e1ab82274e0294a6c12e421a08cd (127.0.97.3:37297): tablet 37b112f1a79b4e2a97fa3ee298f10b94 (table test-table [id=7f3f414b776543a2a5fdba78768c861b]) successfully deleted
I20260927 22:52:02.465318 20235 catalog_manager.cc:4841] TS e9161eadb21949609dc406a60552665b (127.0.97.2:34663): tablet 37b112f1a79b4e2a97fa3ee298f10b94 (table test-table [id=7f3f414b776543a2a5fdba78768c861b]) successfully deleted
I20260927 22:52:02.469849 20741 tablet_replica.cc:330] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: stopping tablet replica
I20260927 22:52:02.470247 20741 raft_consensus.cc:2229] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:52:02.470927 20741 raft_consensus.cc:2258] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:02.472770 20741 ts_tablet_manager.cc:1905] T 37b112f1a79b4e2a97fa3ee298f10b94 P e5dce0ce393142b9a7ea9ec2cdae9436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:02.473925 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.442027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.0.97.1:43923)
0927 22:52:02.442196 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.442202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.464731 (+ 22529us) server_negotiation.cc:407] Connection header received
0927 22:52:02.465700 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.465711 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.465860 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.465977 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:02.466549 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.467906 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.469865 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.470595 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.470748 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.472196 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:02.472208 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:02.472225 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:02.473009 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:02.473085 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:02.473112 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:02.473256 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.473380 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.473491 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:52:02.474707 20432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:02.568738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20310
W20260927 22:52:02.583534 20236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43923 (error 108)
I20260927 22:52:02.583748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20441
I20260927 22:52:02.585443 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.584969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35278)
0927 22:52:02.585121 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.585236 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 22:52:02.585700 20236 catalog_manager.cc:4584] TS e5dce0ce393142b9a7ea9ec2cdae9436 (127.0.97.1:43923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37b112f1a79b4e2a97fa3ee298f10b94: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
I20260927 22:52:02.590835 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.584643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.97.2:34663)
0927 22:52:02.584829 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.584835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.584858 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:52:02.585021 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.585026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.585153 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.585243 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:02.585792 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.586603 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.588096 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.588482 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.588567 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.589597 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:02.589605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:02.589619 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:02.590177 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:02.590223 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:02.590240 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:02.590320 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.590464 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.590545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:52:02.591544 20563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:02.639456 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.638883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35286)
0927 22:52:02.639123 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.639239 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:52:02.640472 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.640081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35296)
0927 22:52:02.640225 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.640327 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 22:52:02.680058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20441
I20260927 22:52:02.693391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20571
I20260927 22:52:02.699697 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.694182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.0.97.3:37297)
0927 22:52:02.694349 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.694353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.694372 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:52:02.694485 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.694489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.694584 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.694664 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:02.694984 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.695759 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.696951 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.697356 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.697454 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.698334 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:02.698342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:02.698357 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:02.698962 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:02.699010 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:02.699029 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:02.699115 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.699215 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.699301 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:52:02.700415 20693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:02.700744 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.700320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35302)
0927 22:52:02.700519 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.700604 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:52:02.701771 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.701411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35314)
0927 22:52:02.701560 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.701641 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 22:52:02.782433 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.781966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35316)
0927 22:52:02.782174 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.782268 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:52:02.783519 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.783114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:35320)
0927 22:52:02.783305 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.783384 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 22:52:02.789634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20571
I20260927 22:52:02.803642 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20220
I20260927 22:52:02.804649 20281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:02.911279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20220
2026-09-27T22:52:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2635 ms)
[----------] 6 tests from EncryptionPolicyTest (21152 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:03 dist-test-slave-kr25 krb5kdc[20751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:03 dist-test-slave-kr25 krb5kdc[20751](info): set up 2 sockets
Sep 27 22:52:03 dist-test-slave-kr25 krb5kdc[20751](info): commencing operation
krb5kdc: starting...
W20260927 22:52:03.939646 388 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:04 dist-test-slave-kr25 krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:04.168442 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:04.484391 20764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:04.484736 20764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:04.484894 20764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:04.492416 20764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:04.492569 20764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:04.492661 20764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:04.492828 20764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:04.492959 20764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:04.503136 20764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:04.504492 20764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:04.505848 20764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:04.517104 20769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:04.517100 20772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:04.517138 20770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:04.517896 20764 server_base.cc:1034] running on GCE node
I20260927 22:52:04.518608 20764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:04.519977 20764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:04.521201 20764 hybrid_clock.cc:648] HybridClock initialized: now 1790549524521138 us; error 43 us; skew 500 ppm
I20260927 22:52:04.521549 20764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:04 dist-test-slave-kr25 krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:04.524484 20764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:04.525909 20764 webserver.cc:460] Webserver started at http://127.0.0.1:41497/ using document root <none> and password file <none>
I20260927 22:52:04.526558 20764 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:04.526677 20764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:04.526971 20764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:04.530200 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "86f9c3b1095e4ddb97e50f8e687c60e2"
format_stamp: "Formatted at 2026-09-27 22:52:04 on dist-test-slave-kr25"
server_key: "47f5acd2f183b28ae3b48d4041cba8af"
server_key_iv: "b32e61996cca8907b893ac39184f055a"
server_key_version: "encryptionkey@0"
I20260927 22:52:04.530973 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "86f9c3b1095e4ddb97e50f8e687c60e2"
format_stamp: "Formatted at 2026-09-27 22:52:04 on dist-test-slave-kr25"
server_key: "47f5acd2f183b28ae3b48d4041cba8af"
server_key_iv: "b32e61996cca8907b893ac39184f055a"
server_key_version: "encryptionkey@0"
I20260927 22:52:04.536222 20764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:52:04.539664 20779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:04.541067 20764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:04.541239 20764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "86f9c3b1095e4ddb97e50f8e687c60e2"
format_stamp: "Formatted at 2026-09-27 22:52:04 on dist-test-slave-kr25"
server_key: "47f5acd2f183b28ae3b48d4041cba8af"
server_key_iv: "b32e61996cca8907b893ac39184f055a"
server_key_version: "encryptionkey@0"
I20260927 22:52:04.541428 20764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:04.568440 20764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:04.572213 20764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:04.572474 20764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:04.588815 20764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37571
I20260927 22:52:04.588848 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37571 every 8 connection(s)
I20260927 22:52:04.590196 20764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:04.593817 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:04.600163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20764
I20260927 22:52:04.600338 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:04.600957 388 external_mini_cluster.cc:1428] Setting key 6ddf86f8dba998a0c99ea76a6be18285
I20260927 22:52:04.604180 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2: Bootstrap starting.
I20260927 22:52:04.607577 20831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:04.608691 20831 log.cc:824] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:04.611555 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2: No bootstrap required, opened a new log
Sep 27 22:52:04 dist-test-slave-kr25 krb5kdc[20751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:52:04.620229 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37571 } }
I20260927 22:52:04.620657 20831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:04.620770 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f9c3b1095e4ddb97e50f8e687c60e2, State: Initialized, Role: FOLLOWER
I20260927 22:52:04.621380 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37571 } }
I20260927 22:52:04.621626 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:04.621776 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:04.621954 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:04.623724 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:04.602819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36543 (local address 127.0.0.1:37571)
0927 22:52:04.603571 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:04.603583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:04.603623 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:52:04.605303 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:04.605325 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:04.605732 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:04.606041 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:04.607803 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:04.609199 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:04.610795 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:04.611490 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:04.611633 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:04.616358 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:04.616369 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:04.616384 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:04.616420 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:04.619411 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:04.620285 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:04.620292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:04.620299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:04.620473 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:04.620933 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:04.620938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:04.620941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:04.621520 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:04.621820 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:04.622289 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 22:52:04.622610 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":320,"threads_started":1}
I20260927 22:52:04.624454 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37571 } }
I20260927 22:52:04.624912 20831 leader_election.cc:302] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f9c3b1095e4ddb97e50f8e687c60e2; no voters:
I20260927 22:52:04.625510 20831 leader_election.cc:288] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:04.625604 20836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:04.626536 20836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [term 1 LEADER]: Becoming Leader. State: Replica: 86f9c3b1095e4ddb97e50f8e687c60e2, State: Running, Role: LEADER
I20260927 22:52:04.627102 20836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37571 } }
I20260927 22:52:04.627754 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:04.627743 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:04.630586 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37571 } } }
I20260927 22:52:04.630612 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f9c3b1095e4ddb97e50f8e687c60e2. Latest consensus state: current_term: 1 leader_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9c3b1095e4ddb97e50f8e687c60e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37571 } } }
I20260927 22:52:04.631146 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:04.631028 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:04.638144 20844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:04.646026 20844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:04.658100 20844 catalog_manager.cc:1347] Generated new cluster ID: b8580d327ab748a685ae748dc504c900
I20260927 22:52:04.658214 20844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:04.674947 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:04.642099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55163 (local address 127.0.0.1:37571)
0927 22:52:04.642254 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:04.642260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:04.657558 (+ 15298us) server_negotiation.cc:407] Connection header received
0927 22:52:04.658091 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:04.658098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:04.658188 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:04.658272 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:04.660783 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:04.661776 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:04.663616 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:04.664299 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:04.664396 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:04.667720 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:04.667727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:04.667731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:04.667771 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:04.671324 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:04.672332 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:04.672337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:04.672341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:04.672510 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:04.673093 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:04.673098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:04.673101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:04.673536 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:04.673791 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:04.674524 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 22:52:04.674674 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 22:52:04.683943 20844 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:04.685161 20844 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:04.690887 20844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f9c3b1095e4ddb97e50f8e687c60e2: Generated new TSK 0
I20260927 22:52:04.691655 20844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:04.699044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20764
I20260927 22:52:04.705001 20826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:04.792281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20764
W20260927 22:52:04.805939 20749 connection.cc:535] client connection to 127.0.0.1:37571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37571): Connection reset by peer (error 104)
2026-09-27T22:52:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:04 dist-test-slave-kr25 krb5kdc[20863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:04 dist-test-slave-kr25 krb5kdc[20863](info): set up 2 sockets
Sep 27 22:52:04 dist-test-slave-kr25 krb5kdc[20863](info): commencing operation
krb5kdc: starting...
W20260927 22:52:05.791391 388 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:05 dist-test-slave-kr25 krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:06.016855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:06.328960 20876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:06.329244 20876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:06.329384 20876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:06.336506 20876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:06.336639 20876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:06.336719 20876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:06.336799 20876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:06.336860 20876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:06.346832 20876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:06.348126 20876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:06.349431 20876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:06.360394 20884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:06.360437 20882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:06.360502 20881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:06.361150 20876 server_base.cc:1034] running on GCE node
I20260927 22:52:06.361848 20876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:06.363161 20876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:06.364357 20876 hybrid_clock.cc:648] HybridClock initialized: now 1790549526364315 us; error 33 us; skew 500 ppm
I20260927 22:52:06.364703 20876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:06 dist-test-slave-kr25 krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:06.367604 20876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:06.369031 20876 webserver.cc:460] Webserver started at http://127.0.0.1:36431/ using document root <none> and password file <none>
I20260927 22:52:06.369680 20876 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:06.369794 20876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:06.370082 20876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:06.373104 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "9f88909462d34a7bae02be28534bc8cb"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-kr25"
server_key: "579bf5ed125d2481944b68b8de431254"
server_key_iv: "f4df4ff5720c4b014e3aedcd8e865ce9"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.373844 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "9f88909462d34a7bae02be28534bc8cb"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-kr25"
server_key: "579bf5ed125d2481944b68b8de431254"
server_key_iv: "f4df4ff5720c4b014e3aedcd8e865ce9"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.378893 20876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:52:06.382351 20891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:06.383735 20876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:06.383894 20876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "9f88909462d34a7bae02be28534bc8cb"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-kr25"
server_key: "579bf5ed125d2481944b68b8de431254"
server_key_iv: "f4df4ff5720c4b014e3aedcd8e865ce9"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.384073 20876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:06.412243 20876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:06.417707 20876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:06.417964 20876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:06.433861 20876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43389
I20260927 22:52:06.433887 20942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43389 every 8 connection(s)
I20260927 22:52:06.435199 20876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:06.437490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20876
I20260927 22:52:06.437670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:06.438190 388 external_mini_cluster.cc:1428] Setting key 7db1dfc738770eabbe614292f469387e
I20260927 22:52:06.439466 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:06.451053 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.440012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49591 (local address 127.0.0.1:43389)
0927 22:52:06.440714 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.440727 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.440772 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:52:06.442291 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.442311 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.442706 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.443019 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.443546 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.444868 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.446707 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.447376 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.447541 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.448786 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:06.448793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:06.448807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:06.449375 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:06.449454 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:06.449483 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:06.449601 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.449935 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.450121 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":315,"threads_started":1}
I20260927 22:52:06.452080 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb: Bootstrap starting.
I20260927 22:52:06.454842 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:06.455624 20943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:06.456817 20943 log.cc:824] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:06.459687 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb: No bootstrap required, opened a new log
I20260927 22:52:06.467576 20943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88909462d34a7bae02be28534bc8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260927 22:52:06.468116 20943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:06.468217 20943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88909462d34a7bae02be28534bc8cb, State: Initialized, Role: FOLLOWER
I20260927 22:52:06.468848 20943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88909462d34a7bae02be28534bc8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260927 22:52:06.469111 20943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:06.469255 20943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:06.469451 20943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:06.469815 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.459781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34595 (local address 127.0.0.1:43389)
0927 22:52:06.459946 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.459951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.463954 (+ 4003us) server_negotiation.cc:407] Connection header received
0927 22:52:06.464119 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.464128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.464204 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.464334 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.464787 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.465919 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.467197 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.467846 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.467970 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.469067 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:06.469074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:06.469078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:06.469104 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:06.469148 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:06.469168 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:06.469305 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.469457 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.469511 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:52:06.472259 20943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88909462d34a7bae02be28534bc8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260927 22:52:06.472709 20943 leader_election.cc:302] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f88909462d34a7bae02be28534bc8cb; no voters:
I20260927 22:52:06.473218 20943 leader_election.cc:288] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:06.473341 20953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:06.474655 20953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [term 1 LEADER]: Becoming Leader. State: Replica: 9f88909462d34a7bae02be28534bc8cb, State: Running, Role: LEADER
I20260927 22:52:06.475238 20943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:06.475191 20953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88909462d34a7bae02be28534bc8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260927 22:52:06.477924 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f88909462d34a7bae02be28534bc8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88909462d34a7bae02be28534bc8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } } }
I20260927 22:52:06.477885 20955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f88909462d34a7bae02be28534bc8cb. Latest consensus state: current_term: 1 leader_uuid: "9f88909462d34a7bae02be28534bc8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88909462d34a7bae02be28534bc8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } } }
I20260927 22:52:06.478327 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:06.478327 20955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:06.479758 20962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:06.484439 20962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:06.493048 20962 catalog_manager.cc:1347] Generated new cluster ID: 754d45a0ed664ea3bcd54c9b8c7b1509
I20260927 22:52:06.493172 20962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:06.519987 20962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:06.521240 20962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:06.537804 20962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f88909462d34a7bae02be28534bc8cb: Generated new TSK 0
I20260927 22:52:06.538851 20962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:06.555248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20876
W20260927 22:52:06.556579 20938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49591
W20260927 22:52:06.557277 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:06.557453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20876
W20260927 22:52:06.570744 20861 connection.cc:535] client connection to 127.0.0.1:43389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43389): Connection reset by peer (error 104)
2026-09-27T22:52:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1764 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:06 dist-test-slave-kr25 krb5kdc[20974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:06 dist-test-slave-kr25 krb5kdc[20974](info): set up 2 sockets
Sep 27 22:52:06 dist-test-slave-kr25 krb5kdc[20974](info): commencing operation
krb5kdc: starting...
W20260927 22:52:07.567376 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:07 dist-test-slave-kr25 krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:07.790553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:08.105412 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:08.105721 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:08.105870 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:08.113042 20987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:08.113168 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:08.113247 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:08.113332 20987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:08.113392 20987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:08.123071 20987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:08.124346 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:08.125608 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:08.136785 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:08.136777 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:08.137029 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:08.137502 20987 server_base.cc:1034] running on GCE node
I20260927 22:52:08.138259 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:08.139624 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:08.140821 20987 hybrid_clock.cc:648] HybridClock initialized: now 1790549528140778 us; error 39 us; skew 500 ppm
I20260927 22:52:08.141152 20987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:08 dist-test-slave-kr25 krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:08.151803 20987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:08.153990 20987 webserver.cc:460] Webserver started at http://127.0.0.1:44585/ using document root <none> and password file <none>
I20260927 22:52:08.154947 20987 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:08.155088 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:08.155427 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:08.159569 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "2c325545adfd4d25919f9a049356ae74"
format_stamp: "Formatted at 2026-09-27 22:52:08 on dist-test-slave-kr25"
server_key: "95bee61a429829295cd1080189d02445"
server_key_iv: "e2148fd1a333bb9629a1e5bdf25254a4"
server_key_version: "encryptionkey@0"
I20260927 22:52:08.160437 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c325545adfd4d25919f9a049356ae74"
format_stamp: "Formatted at 2026-09-27 22:52:08 on dist-test-slave-kr25"
server_key: "95bee61a429829295cd1080189d02445"
server_key_iv: "e2148fd1a333bb9629a1e5bdf25254a4"
server_key_version: "encryptionkey@0"
I20260927 22:52:08.166523 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:52:08.170760 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:08.172287 20987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:08.172492 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "2c325545adfd4d25919f9a049356ae74"
format_stamp: "Formatted at 2026-09-27 22:52:08 on dist-test-slave-kr25"
server_key: "95bee61a429829295cd1080189d02445"
server_key_iv: "e2148fd1a333bb9629a1e5bdf25254a4"
server_key_version: "encryptionkey@0"
I20260927 22:52:08.172680 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:08.178457 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:08.182312 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:08.182555 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:08.199092 20987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41761
I20260927 22:52:08.199133 21053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41761 every 8 connection(s)
I20260927 22:52:08.200493 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:08.204123 21054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:08.210457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 20987
I20260927 22:52:08.210666 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:08.211223 388 external_mini_cluster.cc:1428] Setting key bf94cc3068b2030376fb222ba3fa0e6f
I20260927 22:52:08.214946 21054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74: Bootstrap starting.
I20260927 22:52:08.219033 21054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:08.220647 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.213099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47635 (local address 127.0.0.1:41761)
0927 22:52:08.213856 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.213868 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.213906 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:52:08.215693 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.215730 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.216252 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.216719 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:08.218331 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:08.218340 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:08.218360 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:08.218379 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:08.218469 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:08.218500 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:08.218688 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.218836 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.219239 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":327,"threads_started":1}
I20260927 22:52:08.220983 21054 log.cc:824] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:08.224614 21054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74: No bootstrap required, opened a new log
I20260927 22:52:08.225129 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:08.233420 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.230843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38285 (local address 127.0.0.1:41761)
0927 22:52:08.231029 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.231035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.231503 (+ 468us) server_negotiation.cc:407] Connection header received
0927 22:52:08.231717 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.231722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.231783 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.231895 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:08.232806 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:08.232811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:08.232817 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:08.232838 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:08.232887 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:08.232906 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:08.233028 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.233072 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.233126 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:52:08.235049 21054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c325545adfd4d25919f9a049356ae74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260927 22:52:08.235394 21054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:08.235538 21054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c325545adfd4d25919f9a049356ae74, State: Initialized, Role: FOLLOWER
I20260927 22:52:08.236097 21054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c325545adfd4d25919f9a049356ae74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260927 22:52:08.236312 21054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:08.236433 21054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:08.236583 21054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:08.238709 21054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c325545adfd4d25919f9a049356ae74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260927 22:52:08.239187 21054 leader_election.cc:302] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c325545adfd4d25919f9a049356ae74; no voters:
I20260927 22:52:08.239791 21054 leader_election.cc:288] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:08.239884 21064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:08.240501 21064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [term 1 LEADER]: Becoming Leader. State: Replica: 2c325545adfd4d25919f9a049356ae74, State: Running, Role: LEADER
I20260927 22:52:08.241024 21064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c325545adfd4d25919f9a049356ae74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260927 22:52:08.241766 21054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:08.243752 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c325545adfd4d25919f9a049356ae74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c325545adfd4d25919f9a049356ae74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } } }
I20260927 22:52:08.244163 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:08.244619 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c325545adfd4d25919f9a049356ae74. Latest consensus state: current_term: 1 leader_uuid: "2c325545adfd4d25919f9a049356ae74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c325545adfd4d25919f9a049356ae74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } } }
I20260927 22:52:08.244884 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:08.247967 21071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:08.253270 21071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:08.261853 21071 catalog_manager.cc:1347] Generated new cluster ID: 96d538670ff84aedbf3b15025ffbd048
I20260927 22:52:08.261966 21071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:08.293771 21071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:08.294271 21071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:08.306429 21071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c325545adfd4d25919f9a049356ae74: Generated new TSK 0
I20260927 22:52:08.307559 21071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:08.319633 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20987
W20260927 22:52:08.320644 21049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47635
W20260927 22:52:08.321280 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 20987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:08.321437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 20987
2026-09-27T22:52:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1764 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:08 dist-test-slave-kr25 krb5kdc[21085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:08 dist-test-slave-kr25 krb5kdc[21085](info): set up 2 sockets
Sep 27 22:52:08 dist-test-slave-kr25 krb5kdc[21085](info): commencing operation
krb5kdc: starting...
W20260927 22:52:09.327448 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:09 dist-test-slave-kr25 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:09.556200 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:09.863375 21098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:09.863659 21098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:09.863758 21098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:09.870821 21098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:09.870941 21098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:09.871002 21098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:09.871062 21098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:09.871105 21098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:09.880719 21098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:09.881919 21098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:09.883164 21098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:09.894086 21103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:09.894129 21104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:09.894053 21106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:09.894727 21098 server_base.cc:1034] running on GCE node
I20260927 22:52:09.895498 21098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:09.896852 21098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:09.898043 21098 hybrid_clock.cc:648] HybridClock initialized: now 1790549529897995 us; error 41 us; skew 500 ppm
I20260927 22:52:09.898384 21098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:09 dist-test-slave-kr25 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:09.901311 21098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:09.902688 21098 webserver.cc:460] Webserver started at http://127.0.0.1:36381/ using document root <none> and password file <none>
I20260927 22:52:09.903329 21098 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:09.903460 21098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:09.903759 21098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:09.906646 21098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "9558f18f292348c198cfc9544a84649d"
format_stamp: "Formatted at 2026-09-27 22:52:09 on dist-test-slave-kr25"
server_key: "8f195e8d595b81ed4de3020a3089a709"
server_key_iv: "38a8024886db5121e137ad786894d807"
server_key_version: "encryptionkey@0"
I20260927 22:52:09.907361 21098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "9558f18f292348c198cfc9544a84649d"
format_stamp: "Formatted at 2026-09-27 22:52:09 on dist-test-slave-kr25"
server_key: "8f195e8d595b81ed4de3020a3089a709"
server_key_iv: "38a8024886db5121e137ad786894d807"
server_key_version: "encryptionkey@0"
I20260927 22:52:09.912218 21098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 22:52:09.915391 21113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:09.916653 21098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:09.916800 21098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "9558f18f292348c198cfc9544a84649d"
format_stamp: "Formatted at 2026-09-27 22:52:09 on dist-test-slave-kr25"
server_key: "8f195e8d595b81ed4de3020a3089a709"
server_key_iv: "38a8024886db5121e137ad786894d807"
server_key_version: "encryptionkey@0"
I20260927 22:52:09.916963 21098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:09.933184 21098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:09.937099 21098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:09.937336 21098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:09.952764 21098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35267
I20260927 22:52:09.952792 21164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35267 every 8 connection(s)
I20260927 22:52:09.954036 21098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:09.955482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21098
I20260927 22:52:09.955651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:09.956172 388 external_mini_cluster.cc:1428] Setting key a53374a77371abc767c928201aa38d23
I20260927 22:52:09.958393 21165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:52:09 dist-test-slave-kr25 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:52:09.970336 21165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d: Bootstrap starting.
I20260927 22:52:09.974056 21165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:09.975294 21165 log.cc:824] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:09.978633 21165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d: No bootstrap required, opened a new log
I20260927 22:52:09.979151 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:09.957842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43025 (local address 127.0.0.1:35267)
0927 22:52:09.958564 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:09.958578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:09.958624 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:52:09.960857 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:09.960883 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:09.961377 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:09.961796 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:09.963021 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:09.965009 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:09.966215 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:09.967175 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:09.967363 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:09.971115 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:09.971125 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:09.971143 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:09.971189 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:09.974850 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:09.975528 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:09.975536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:09.975544 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:09.975736 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:09.976181 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:09.976184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:09.976187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:09.976778 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:09.977044 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:09.977409 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 22:52:09.977857 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":297,"threads_started":1}
I20260927 22:52:09.983104 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:09.988466 21165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9558f18f292348c198cfc9544a84649d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260927 22:52:09.988833 21165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:09.988930 21165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9558f18f292348c198cfc9544a84649d, State: Initialized, Role: FOLLOWER
I20260927 22:52:09.989472 21165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9558f18f292348c198cfc9544a84649d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260927 22:52:09.989674 21165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:09.989794 21165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:09.989934 21165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:09.992130 21165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9558f18f292348c198cfc9544a84649d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260927 22:52:09.992592 21165 leader_election.cc:302] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9558f18f292348c198cfc9544a84649d; no voters:
I20260927 22:52:09.993207 21165 leader_election.cc:288] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:09.993338 21175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:09.994067 21175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [term 1 LEADER]: Becoming Leader. State: Replica: 9558f18f292348c198cfc9544a84649d, State: Running, Role: LEADER
I20260927 22:52:09.994679 21175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9558f18f292348c198cfc9544a84649d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260927 22:52:09.995399 21165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:09.997493 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9558f18f292348c198cfc9544a84649d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9558f18f292348c198cfc9544a84649d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } } }
I20260927 22:52:09.997859 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:09.998335 21177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9558f18f292348c198cfc9544a84649d. Latest consensus state: current_term: 1 leader_uuid: "9558f18f292348c198cfc9544a84649d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9558f18f292348c198cfc9544a84649d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } } }
I20260927 22:52:09.998639 21177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:10.002643 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:10.003099 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:09.988173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42779 (local address 127.0.0.1:35267)
0927 22:52:09.988353 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:09.988359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:09.988736 (+ 377us) server_negotiation.cc:407] Connection header received
0927 22:52:09.988907 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:09.988913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:09.988987 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:09.989068 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:09.990735 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:09.991490 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:09.992731 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:09.993192 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:09.993302 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:09.995865 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:09.995870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:09.995873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:09.995906 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:09.999602 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:10.000605 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:10.000611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:10.000615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:10.000839 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:10.001291 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:10.001297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:10.001300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:10.001696 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:10.001998 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:10.002298 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:52:10.002513 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:52:10.011209 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:10.020152 21180 catalog_manager.cc:1347] Generated new cluster ID: 2c6ffa2b32e04ddea2e9a5bd7ac03841
I20260927 22:52:10.020272 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:10.041024 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:10.042917 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:10.058275 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9558f18f292348c198cfc9544a84649d: Generated new TSK 0
I20260927 22:52:10.059466 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:10.071621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21098
I20260927 22:52:10.072841 21160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:10.164256 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21098
2026-09-27T22:52:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:10 dist-test-slave-kr25 krb5kdc[21197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:10 dist-test-slave-kr25 krb5kdc[21197](info): set up 2 sockets
Sep 27 22:52:10 dist-test-slave-kr25 krb5kdc[21197](info): commencing operation
krb5kdc: starting...
W20260927 22:52:11.179498 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.007s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:11 dist-test-slave-kr25 krb5kdc[21197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:11.410369 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:11.722016 21210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:11.722301 21210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:11.722430 21210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:11.729431 21210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:11.729530 21210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:11.729607 21210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:11.729682 21210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:11.729737 21210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:11.741817 21210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:11.743077 21210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:11.744376 21210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:11.755119 21215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:11.755162 21218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:11.755193 21216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:11.756260 21210 server_base.cc:1034] running on GCE node
I20260927 22:52:11.756938 21210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:11.758194 21210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:11.759392 21210 hybrid_clock.cc:648] HybridClock initialized: now 1790549531759342 us; error 42 us; skew 500 ppm
I20260927 22:52:11.759711 21210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:11 dist-test-slave-kr25 krb5kdc[21197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:11.762449 21210 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:11.763796 21210 webserver.cc:460] Webserver started at http://127.0.0.1:36349/ using document root <none> and password file <none>
I20260927 22:52:11.764410 21210 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:11.764519 21210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:11.764787 21210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:11.767534 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "f2d34da928494ed8a237715ba4226ffb"
format_stamp: "Formatted at 2026-09-27 22:52:11 on dist-test-slave-kr25"
server_key: "c0b5a72f7653b3035b8ed00ddf7c2321"
server_key_iv: "3ec76351b8fab9d5f68773279d929d50"
server_key_version: "encryptionkey@0"
I20260927 22:52:11.768227 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "f2d34da928494ed8a237715ba4226ffb"
format_stamp: "Formatted at 2026-09-27 22:52:11 on dist-test-slave-kr25"
server_key: "c0b5a72f7653b3035b8ed00ddf7c2321"
server_key_iv: "3ec76351b8fab9d5f68773279d929d50"
server_key_version: "encryptionkey@0"
I20260927 22:52:11.772807 21210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:52:11.775867 21225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:11.777048 21210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:52:11.777204 21210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "f2d34da928494ed8a237715ba4226ffb"
format_stamp: "Formatted at 2026-09-27 22:52:11 on dist-test-slave-kr25"
server_key: "c0b5a72f7653b3035b8ed00ddf7c2321"
server_key_iv: "3ec76351b8fab9d5f68773279d929d50"
server_key_version: "encryptionkey@0"
I20260927 22:52:11.777379 21210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:11.789124 21210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:11.792973 21210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:11.793222 21210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:11.809741 21210 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41713
I20260927 22:52:11.809764 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41713 every 8 connection(s)
I20260927 22:52:11.811079 21210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:11.811458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21210
I20260927 22:52:11.811607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:11.812103 388 external_mini_cluster.cc:1428] Setting key ea9f8d055c79992971a4fa27f556090b
I20260927 22:52:11.815107 21277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:11.827098 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:11.813915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40885 (local address 127.0.0.1:41713)
0927 22:52:11.814766 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:11.814782 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:11.815127 (+ 345us) server_negotiation.cc:407] Connection header received
0927 22:52:11.817392 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:11.817421 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:11.817922 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:11.818368 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:11.818870 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:11.820846 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:11.822131 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:11.823130 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:11.823313 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:11.824371 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:11.824381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:11.824399 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:11.825113 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:11.825209 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:11.825248 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:11.825363 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:11.825561 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 22:52:11.825839 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":365,"threads_started":1}
I20260927 22:52:11.828436 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb: Bootstrap starting.
I20260927 22:52:11.830756 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:11.831809 21277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:11.832705 21277 log.cc:824] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:11.835217 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb: No bootstrap required, opened a new log
I20260927 22:52:11.840647 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:11.834780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50745 (local address 127.0.0.1:41713)
0927 22:52:11.834941 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:11.834947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:11.835314 (+ 367us) server_negotiation.cc:407] Connection header received
0927 22:52:11.835503 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:11.835508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:11.835579 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:11.835695 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:11.836090 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:11.837126 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:11.838326 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:11.838990 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:11.839111 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:11.839991 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:11.839998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:11.840002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:11.840024 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:11.840069 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:11.840087 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:11.840176 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:11.840264 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 22:52:11.840352 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:52:11.843667 21277 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d34da928494ed8a237715ba4226ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260927 22:52:11.843995 21277 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:11.844082 21277 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d34da928494ed8a237715ba4226ffb, State: Initialized, Role: FOLLOWER
I20260927 22:52:11.844615 21277 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d34da928494ed8a237715ba4226ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260927 22:52:11.844882 21277 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:11.845006 21277 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:11.845158 21277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:11.847139 21277 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d34da928494ed8a237715ba4226ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260927 22:52:11.847621 21277 leader_election.cc:302] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2d34da928494ed8a237715ba4226ffb; no voters:
I20260927 22:52:11.848129 21277 leader_election.cc:288] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:11.848229 21287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:11.848891 21287 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [term 1 LEADER]: Becoming Leader. State: Replica: f2d34da928494ed8a237715ba4226ffb, State: Running, Role: LEADER
I20260927 22:52:11.849465 21287 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d34da928494ed8a237715ba4226ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260927 22:52:11.849944 21277 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:11.852415 21289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2d34da928494ed8a237715ba4226ffb. Latest consensus state: current_term: 1 leader_uuid: "f2d34da928494ed8a237715ba4226ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d34da928494ed8a237715ba4226ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } } }
I20260927 22:52:11.852427 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2d34da928494ed8a237715ba4226ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d34da928494ed8a237715ba4226ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } } }
I20260927 22:52:11.852838 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:11.852840 21289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:11.854146 21296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:11.859074 21296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:11.867880 21296 catalog_manager.cc:1347] Generated new cluster ID: 84c38532ab044fd892d2e92fb225a5d8
I20260927 22:52:11.867988 21296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:11.903859 21296 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:11.905226 21296 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:11.915372 21296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2d34da928494ed8a237715ba4226ffb: Generated new TSK 0
I20260927 22:52:11.916200 21296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:11.922168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21210
W20260927 22:52:11.923317 21272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40885
W20260927 22:52:11.923960 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:11.924109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21210
2026-09-27T22:52:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1761 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:12 dist-test-slave-kr25 krb5kdc[21308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:12 dist-test-slave-kr25 krb5kdc[21308](info): set up 2 sockets
Sep 27 22:52:12 dist-test-slave-kr25 krb5kdc[21308](info): commencing operation
krb5kdc: starting...
W20260927 22:52:12.923484 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:13 dist-test-slave-kr25 krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:13.151767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:13.479888 21321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:13.480170 21321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:13.480295 21321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:13.487552 21321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:13.487669 21321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:13.487741 21321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:13.487828 21321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:13.487884 21321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:13.497835 21321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:13.499138 21321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:13.500460 21321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:13.512025 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:13.511996 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:13.512251 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:13.512912 21321 server_base.cc:1034] running on GCE node
I20260927 22:52:13.513677 21321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:13.515121 21321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:13.516316 21321 hybrid_clock.cc:648] HybridClock initialized: now 1790549533516258 us; error 53 us; skew 500 ppm
I20260927 22:52:13.516638 21321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:13 dist-test-slave-kr25 krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:13.519559 21321 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:13.520913 21321 webserver.cc:460] Webserver started at http://127.0.0.1:40607/ using document root <none> and password file <none>
I20260927 22:52:13.521562 21321 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:13.521677 21321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:13.521960 21321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:13.524914 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "840cabefece5437884de68dc1b25f255"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-kr25"
server_key: "d6625dfc995e2d76d86039e56d470337"
server_key_iv: "d3d4bb3c6a0a0080412a1749ebb52a46"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.525629 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "840cabefece5437884de68dc1b25f255"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-kr25"
server_key: "d6625dfc995e2d76d86039e56d470337"
server_key_iv: "d3d4bb3c6a0a0080412a1749ebb52a46"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.530529 21321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:52:13.533820 21336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:13.535110 21321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:52:13.535261 21321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "840cabefece5437884de68dc1b25f255"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-kr25"
server_key: "d6625dfc995e2d76d86039e56d470337"
server_key_iv: "d3d4bb3c6a0a0080412a1749ebb52a46"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.535475 21321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:13.540868 21321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:13.545094 21321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:13.545365 21321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:13.562326 21321 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40959
I20260927 22:52:13.562369 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40959 every 8 connection(s)
I20260927 22:52:13.563695 21321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:13.567257 21388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:13.573887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21321
I20260927 22:52:13.574043 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:13.574482 388 external_mini_cluster.cc:1428] Setting key fc4877d6b374075cf24a13cf476d291d
I20260927 22:52:13.577338 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255: Bootstrap starting.
I20260927 22:52:13.580646 21388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:13.581751 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:13.576114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34793 (local address 127.0.0.1:40959)
0927 22:52:13.576876 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:13.576888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:13.576929 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:52:13.578596 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:13.578623 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:13.578973 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:13.579286 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:13.580288 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:13.580294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:13.580307 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:13.580326 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:13.580381 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:13.580400 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:13.580534 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:13.580680 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:52:13.580872 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":321,"threads_started":1}
I20260927 22:52:13.581919 21388 log.cc:824] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:13.585104 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255: No bootstrap required, opened a new log
I20260927 22:52:13.585582 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:13.591759 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:13.589538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59223 (local address 127.0.0.1:40959)
0927 22:52:13.589731 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:13.589736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:13.590052 (+ 316us) server_negotiation.cc:407] Connection header received
0927 22:52:13.590196 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:13.590201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:13.590258 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:13.590368 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:13.591117 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:13.591123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:13.591131 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:13.591149 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:13.591207 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:13.591227 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:13.591347 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:13.591388 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 22:52:13.591464 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:52:13.593235 21388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840cabefece5437884de68dc1b25f255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260927 22:52:13.593575 21388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:13.593667 21388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840cabefece5437884de68dc1b25f255, State: Initialized, Role: FOLLOWER
I20260927 22:52:13.594199 21388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840cabefece5437884de68dc1b25f255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260927 22:52:13.594429 21388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:13.594558 21388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:13.594710 21388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:13.596874 21388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840cabefece5437884de68dc1b25f255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260927 22:52:13.597344 21388 leader_election.cc:302] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840cabefece5437884de68dc1b25f255; no voters:
I20260927 22:52:13.597934 21388 leader_election.cc:288] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:13.598040 21398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:13.598656 21398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [term 1 LEADER]: Becoming Leader. State: Replica: 840cabefece5437884de68dc1b25f255, State: Running, Role: LEADER
I20260927 22:52:13.599215 21398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840cabefece5437884de68dc1b25f255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260927 22:52:13.600143 21388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:13.602262 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "840cabefece5437884de68dc1b25f255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840cabefece5437884de68dc1b25f255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } } }
I20260927 22:52:13.602635 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:13.602213 21400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840cabefece5437884de68dc1b25f255. Latest consensus state: current_term: 1 leader_uuid: "840cabefece5437884de68dc1b25f255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840cabefece5437884de68dc1b25f255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } } }
I20260927 22:52:13.603134 21400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:13.608168 21404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:13.613214 21404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:13.622848 21404 catalog_manager.cc:1347] Generated new cluster ID: d057f207111141adbc29fef15c49e1c1
I20260927 22:52:13.622967 21404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:13.653118 21404 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:13.653587 21404 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:13.663968 21404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 840cabefece5437884de68dc1b25f255: Generated new TSK 0
I20260927 22:52:13.664840 21404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:13.682632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21321
W20260927 22:52:13.683712 21383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34793
W20260927 22:52:13.684401 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21321: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:13.684530 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21321
2026-09-27T22:52:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1758 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:13 dist-test-slave-kr25 krb5kdc[21419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:13 dist-test-slave-kr25 krb5kdc[21419](info): set up 2 sockets
Sep 27 22:52:13 dist-test-slave-kr25 krb5kdc[21419](info): commencing operation
krb5kdc: starting...
W20260927 22:52:14.687371 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:14 dist-test-slave-kr25 krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:14.915169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:15.230149 21432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:15.230453 21432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:15.230585 21432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:15.237788 21432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:15.237916 21432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:15.237994 21432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:15.238070 21432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:15.238126 21432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:15.247967 21432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:15.249221 21432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:15.250480 21432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:15.261588 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:15.261626 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:15.261646 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:15.261855 21432 server_base.cc:1034] running on GCE node
I20260927 22:52:15.262794 21432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:15.264125 21432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:15.265316 21432 hybrid_clock.cc:648] HybridClock initialized: now 1790549535265265 us; error 37 us; skew 500 ppm
I20260927 22:52:15.265606 21432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:15 dist-test-slave-kr25 krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:15.268498 21432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:15.269868 21432 webserver.cc:460] Webserver started at http://127.0.0.1:36077/ using document root <none> and password file <none>
I20260927 22:52:15.270504 21432 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:15.270617 21432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:15.270895 21432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:15.273734 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "28157438794944fa8ad3dc04ef9651fc"
format_stamp: "Formatted at 2026-09-27 22:52:15 on dist-test-slave-kr25"
server_key: "c3e4d329fab3b43c1063a7082b923705"
server_key_iv: "ca2594d4c2f2841b512cfcc79b9a25dd"
server_key_version: "encryptionkey@0"
I20260927 22:52:15.274449 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "28157438794944fa8ad3dc04ef9651fc"
format_stamp: "Formatted at 2026-09-27 22:52:15 on dist-test-slave-kr25"
server_key: "c3e4d329fab3b43c1063a7082b923705"
server_key_iv: "ca2594d4c2f2841b512cfcc79b9a25dd"
server_key_version: "encryptionkey@0"
I20260927 22:52:15.279143 21432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 22:52:15.282274 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:15.283502 21432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 22:52:15.283656 21432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "28157438794944fa8ad3dc04ef9651fc"
format_stamp: "Formatted at 2026-09-27 22:52:15 on dist-test-slave-kr25"
server_key: "c3e4d329fab3b43c1063a7082b923705"
server_key_iv: "ca2594d4c2f2841b512cfcc79b9a25dd"
server_key_version: "encryptionkey@0"
I20260927 22:52:15.283826 21432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:15.300755 21432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:15.304414 21432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:15.304669 21432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:15.320974 21432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35743
I20260927 22:52:15.320997 21498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35743 every 8 connection(s)
I20260927 22:52:15.322281 21432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:15.325747 21499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:15.328573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21432
I20260927 22:52:15.328766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:15.329258 388 external_mini_cluster.cc:1428] Setting key e9cef903d0999e163a498d2201b81d2f
I20260927 22:52:15.337049 21499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc: Bootstrap starting.
I20260927 22:52:15.340497 21499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:15.341681 21499 log.cc:824] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:52:15 dist-test-slave-kr25 krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:52:15.344686 21499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc: No bootstrap required, opened a new log
I20260927 22:52:15.352365 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.330920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:51807 (local address 127.0.0.1:35743)
0927 22:52:15.331659 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.331670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.331716 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:52:15.333308 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.333329 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.333721 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.334023 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.335739 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.336987 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.338668 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.339306 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.339458 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.343949 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:15.343960 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:15.343979 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:15.344028 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:15.347910 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:15.348674 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:15.348682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:15.348691 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:15.348896 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:15.349276 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:15.349281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:15.349285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:15.349975 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:15.350319 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:15.350606 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:52:15.350967 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":339,"threads_started":1}
I20260927 22:52:15.354488 21499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28157438794944fa8ad3dc04ef9651fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35743 } }
I20260927 22:52:15.354847 21499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:15.354954 21499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157438794944fa8ad3dc04ef9651fc, State: Initialized, Role: FOLLOWER
I20260927 22:52:15.355520 21499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28157438794944fa8ad3dc04ef9651fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35743 } }
I20260927 22:52:15.355760 21499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:15.355890 21499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:15.356024 21499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:15.356053 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:15.358068 21499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28157438794944fa8ad3dc04ef9651fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35743 } }
I20260927 22:52:15.358562 21499 leader_election.cc:302] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28157438794944fa8ad3dc04ef9651fc; no voters:
I20260927 22:52:15.359822 21499 leader_election.cc:288] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:15.359964 21508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:15.361701 21508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [term 1 LEADER]: Becoming Leader. State: Replica: 28157438794944fa8ad3dc04ef9651fc, State: Running, Role: LEADER
I20260927 22:52:15.362020 21499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:15.362310 21508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28157438794944fa8ad3dc04ef9651fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35743 } }
I20260927 22:52:15.366024 21511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28157438794944fa8ad3dc04ef9651fc. Latest consensus state: current_term: 1 leader_uuid: "28157438794944fa8ad3dc04ef9651fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28157438794944fa8ad3dc04ef9651fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35743 } } }
I20260927 22:52:15.366400 21511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:15.366883 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28157438794944fa8ad3dc04ef9651fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28157438794944fa8ad3dc04ef9651fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35743 } } }
I20260927 22:52:15.367151 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:15.369877 21517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:15.375905 21517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:15.378571 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.360003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:52273 (local address 127.0.0.1:35743)
0927 22:52:15.360166 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.360172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.361324 (+ 1152us) server_negotiation.cc:407] Connection header received
0927 22:52:15.361487 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.361492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.361581 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.361679 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.363549 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.364927 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.366655 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.367342 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.367467 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.370595 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:15.370600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:15.370603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:15.370631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:15.374135 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:15.375226 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:15.375232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:15.375236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:15.375386 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:15.376099 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:15.376104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:15.376108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:15.376540 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:15.377780 (+ 1240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:15.377975 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:52:15.378159 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:52:15.387238 21517 catalog_manager.cc:1347] Generated new cluster ID: e94c4bcd3ffb498aa571ec6c715d6e3d
I20260927 22:52:15.387360 21517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:15.394606 21517 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:15.396397 21517 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:15.410740 21517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28157438794944fa8ad3dc04ef9651fc: Generated new TSK 0
I20260927 22:52:15.411818 21517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:15.422642 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21432
I20260927 22:52:15.423877 21494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:15.513015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21432
W20260927 22:52:15.526813 21417 connection.cc:535] client connection to 127.0.0.1:35743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35743): Connection reset by peer (error 104)
2026-09-27T22:52:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1828 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:15 dist-test-slave-kr25 krb5kdc[21532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:15 dist-test-slave-kr25 krb5kdc[21532](info): set up 2 sockets
Sep 27 22:52:15 dist-test-slave-kr25 krb5kdc[21532](info): commencing operation
krb5kdc: starting...
W20260927 22:52:16.523597 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:16 dist-test-slave-kr25 krb5kdc[21532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:16.754431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:17.063242 21545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:17.063568 21545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:17.063727 21545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:17.070885 21545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:17.071005 21545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:17.071079 21545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:17.071162 21545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:17.071223 21545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:17.081066 21545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:17.082288 21545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:17.083552 21545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:17.094452 21550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:17.094370 21551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:17.094416 21553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:17.095247 21545 server_base.cc:1034] running on GCE node
I20260927 22:52:17.096056 21545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:17.097419 21545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:17.098603 21545 hybrid_clock.cc:648] HybridClock initialized: now 1790549537098561 us; error 42 us; skew 500 ppm
I20260927 22:52:17.098927 21545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:17 dist-test-slave-kr25 krb5kdc[21532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:17.101933 21545 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:17.103406 21545 webserver.cc:460] Webserver started at http://127.0.0.1:33311/ using document root <none> and password file <none>
I20260927 22:52:17.104090 21545 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:17.104197 21545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:17.104444 21545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:17.107304 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "95f18157ef5c4661914185542b11211b"
format_stamp: "Formatted at 2026-09-27 22:52:17 on dist-test-slave-kr25"
server_key: "f2e5afb8fd6a0493e7db1545840fcbec"
server_key_iv: "d8ec28ecf5c5a4ae4c59cc2a60615e02"
server_key_version: "encryptionkey@0"
I20260927 22:52:17.108063 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "95f18157ef5c4661914185542b11211b"
format_stamp: "Formatted at 2026-09-27 22:52:17 on dist-test-slave-kr25"
server_key: "f2e5afb8fd6a0493e7db1545840fcbec"
server_key_iv: "d8ec28ecf5c5a4ae4c59cc2a60615e02"
server_key_version: "encryptionkey@0"
I20260927 22:52:17.112937 21545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:52:17.116245 21560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:17.117491 21545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:17.117645 21545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "95f18157ef5c4661914185542b11211b"
format_stamp: "Formatted at 2026-09-27 22:52:17 on dist-test-slave-kr25"
server_key: "f2e5afb8fd6a0493e7db1545840fcbec"
server_key_iv: "d8ec28ecf5c5a4ae4c59cc2a60615e02"
server_key_version: "encryptionkey@0"
I20260927 22:52:17.117823 21545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:17.133000 21545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:17.136981 21545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:17.137224 21545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:17.153919 21545 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32907
I20260927 22:52:17.153952 21611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32907 every 8 connection(s)
I20260927 22:52:17.155259 21545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:17.158841 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:17.164608 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21545
I20260927 22:52:17.164775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:17.165156 388 external_mini_cluster.cc:1428] Setting key d8cf8592d7402eb9cdf13f6fae25e1c6
I20260927 22:52:17.169241 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b: Bootstrap starting.
I20260927 22:52:17.172602 21612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:17.173813 21612 log.cc:824] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:17.176761 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b: No bootstrap required, opened a new log
I20260927 22:52:17.178038 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:17.166823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:56141 (local address 127.0.0.1:32907)
0927 22:52:17.167578 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:17.167590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:17.167630 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:52:17.169254 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:17.169276 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:17.169696 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:17.170000 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:17.170594 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.171935 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.173777 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.174659 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.174849 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.175914 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:17.175921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:17.175938 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:17.176425 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:17.176497 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:17.176520 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:17.176632 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:17.176938 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:52:17.177115 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":328,"threads_started":1}
I20260927 22:52:17.181625 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:17.185644 21612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f18157ef5c4661914185542b11211b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32907 } }
I20260927 22:52:17.186087 21612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:17.186219 21612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f18157ef5c4661914185542b11211b, State: Initialized, Role: FOLLOWER
I20260927 22:52:17.186896 21612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f18157ef5c4661914185542b11211b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32907 } }
I20260927 22:52:17.187189 21612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:17.187321 21612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:17.187500 21612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:17.189710 21612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f18157ef5c4661914185542b11211b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32907 } }
I20260927 22:52:17.190179 21612 leader_election.cc:302] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95f18157ef5c4661914185542b11211b; no voters:
I20260927 22:52:17.190747 21612 leader_election.cc:288] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:17.190953 21622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:17.191843 21622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [term 1 LEADER]: Becoming Leader. State: Replica: 95f18157ef5c4661914185542b11211b, State: Running, Role: LEADER
I20260927 22:52:17.192704 21622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f18157ef5c4661914185542b11211b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32907 } }
I20260927 22:52:17.193231 21612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:17.193521 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:17.186762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:54487 (local address 127.0.0.1:32907)
0927 22:52:17.186917 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:17.186922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:17.187382 (+ 460us) server_negotiation.cc:407] Connection header received
0927 22:52:17.187592 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:17.187597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:17.187665 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:17.187763 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:17.188270 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.189034 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.190350 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.191003 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.191137 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.192647 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:17.192678 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:17.192683 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:17.192710 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:17.192757 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:17.192778 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:17.192896 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:17.193120 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 22:52:17.193172 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:52:17.195660 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95f18157ef5c4661914185542b11211b. Latest consensus state: current_term: 1 leader_uuid: "95f18157ef5c4661914185542b11211b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f18157ef5c4661914185542b11211b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32907 } } }
I20260927 22:52:17.195693 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95f18157ef5c4661914185542b11211b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f18157ef5c4661914185542b11211b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32907 } } }
I20260927 22:52:17.196060 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:17.196060 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:17.196796 21627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:17.201665 21627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:17.211110 21627 catalog_manager.cc:1347] Generated new cluster ID: 8a5bc1c6df7c4ce2b581c23c7c778674
I20260927 22:52:17.211234 21627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:17.233829 21627 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:17.235708 21627 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:17.249838 21627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95f18157ef5c4661914185542b11211b: Generated new TSK 0
I20260927 22:52:17.250828 21627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:17.262300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21545
W20260927 22:52:17.263283 21607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:56141
W20260927 22:52:17.263980 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:17.264135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21545
W20260927 22:52:17.277915 21530 connection.cc:535] client connection to 127.0.0.1:32907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32907): Connection reset by peer (error 104)
2026-09-27T22:52:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1751 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:17 dist-test-slave-kr25 krb5kdc[21643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:17 dist-test-slave-kr25 krb5kdc[21643](info): set up 2 sockets
Sep 27 22:52:17 dist-test-slave-kr25 krb5kdc[21643](info): commencing operation
krb5kdc: starting...
W20260927 22:52:18.275528 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:18 dist-test-slave-kr25 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:18.510710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:18.824910 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:18.825202 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:18.825335 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:18.832481 21656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:18.832605 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:18.832685 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:18.832756 21656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:18.832808 21656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:18.842506 21656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:18.843768 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:18.845053 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:18.856259 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:18.856338 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:18.856290 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:18.857247 21656 server_base.cc:1034] running on GCE node
I20260927 22:52:18.858045 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:18.859454 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:18.860677 21656 hybrid_clock.cc:648] HybridClock initialized: now 1790549538860621 us; error 48 us; skew 500 ppm
I20260927 22:52:18.861032 21656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:18 dist-test-slave-kr25 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:18.867420 21656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:18.868958 21656 webserver.cc:460] Webserver started at http://127.0.0.1:38257/ using document root <none> and password file <none>
I20260927 22:52:18.869645 21656 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:18.869765 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:18.870059 21656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:18.873030 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "57ed3f45d8ae43a8927c1db6384ae48a"
format_stamp: "Formatted at 2026-09-27 22:52:18 on dist-test-slave-kr25"
server_key: "710c7c7bcafe1b798af7a75e5ce96cc2"
server_key_iv: "3f0cba6b4b7e0e672bbc30c0bca04dba"
server_key_version: "encryptionkey@0"
I20260927 22:52:18.873759 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "57ed3f45d8ae43a8927c1db6384ae48a"
format_stamp: "Formatted at 2026-09-27 22:52:18 on dist-test-slave-kr25"
server_key: "710c7c7bcafe1b798af7a75e5ce96cc2"
server_key_iv: "3f0cba6b4b7e0e672bbc30c0bca04dba"
server_key_version: "encryptionkey@0"
I20260927 22:52:18.878719 21656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 22:52:18.881950 21671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:18.883185 21656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:18.883334 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "57ed3f45d8ae43a8927c1db6384ae48a"
format_stamp: "Formatted at 2026-09-27 22:52:18 on dist-test-slave-kr25"
server_key: "710c7c7bcafe1b798af7a75e5ce96cc2"
server_key_iv: "3f0cba6b4b7e0e672bbc30c0bca04dba"
server_key_version: "encryptionkey@0"
I20260927 22:52:18.883548 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:18.888927 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:18.892515 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:18.892765 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:18.909096 21656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35713
I20260927 22:52:18.909132 21722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35713 every 8 connection(s)
I20260927 22:52:18.910429 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:18.911507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21656
I20260927 22:52:18.911688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:18.912184 388 external_mini_cluster.cc:1428] Setting key 5b265651e0d43153a0dd8d7476c346e8
I20260927 22:52:18.914891 21723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:18.919546 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:18.913938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:42981 (local address 127.0.0.1:35713)
0927 22:52:18.914658 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:18.914671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:18.914711 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:52:18.916368 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:18.916390 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:18.916703 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:18.917051 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:18.917988 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:18.917993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:18.918006 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:18.918019 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:18.918098 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:18.918124 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:18.918331 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:18.918461 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 22:52:18.918628 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":302,"threads_started":1}
I20260927 22:52:18.922847 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:18.925701 21723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a: Bootstrap starting.
I20260927 22:52:18.929226 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:18.927090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:42975 (local address 127.0.0.1:35713)
0927 22:52:18.927268 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:18.927272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:18.927501 (+ 229us) server_negotiation.cc:407] Connection header received
0927 22:52:18.927682 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:18.927685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:18.927730 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:18.927802 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:18.928672 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:18.928676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:18.928679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:18.928691 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:18.928726 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:18.928739 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:18.928819 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:18.928927 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 22:52:18.928971 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:52:18.929301 21723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:18.930325 21723 log.cc:824] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:18.932811 21723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a: No bootstrap required, opened a new log
I20260927 22:52:18.939610 21723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260927 22:52:18.939956 21723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:18.940040 21723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ed3f45d8ae43a8927c1db6384ae48a, State: Initialized, Role: FOLLOWER
I20260927 22:52:18.940546 21723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260927 22:52:18.940752 21723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:18.940871 21723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:18.941013 21723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:18.943112 21723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260927 22:52:18.943624 21723 leader_election.cc:302] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ed3f45d8ae43a8927c1db6384ae48a; no voters:
I20260927 22:52:18.944132 21723 leader_election.cc:288] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:18.944245 21733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:18.944828 21733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [term 1 LEADER]: Becoming Leader. State: Replica: 57ed3f45d8ae43a8927c1db6384ae48a, State: Running, Role: LEADER
I20260927 22:52:18.945348 21733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260927 22:52:18.946051 21723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:18.948072 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } } }
I20260927 22:52:18.948107 21735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ed3f45d8ae43a8927c1db6384ae48a. Latest consensus state: current_term: 1 leader_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ed3f45d8ae43a8927c1db6384ae48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } } }
I20260927 22:52:18.948542 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:18.948549 21735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:18.950363 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:18.955621 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:18.963582 21739 catalog_manager.cc:1347] Generated new cluster ID: c3eee0b532b94651b42fdc532958c7ee
I20260927 22:52:18.963694 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:18.980212 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:18.980589 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:18.992239 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57ed3f45d8ae43a8927c1db6384ae48a: Generated new TSK 0
I20260927 22:52:18.993017 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:18.997727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21656
W20260927 22:52:18.998696 21718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:42981
W20260927 22:52:18.999302 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21656: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:18.999420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21656
2026-09-27T22:52:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1734 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:19 dist-test-slave-kr25 krb5kdc[21754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:19 dist-test-slave-kr25 krb5kdc[21754](info): set up 2 sockets
Sep 27 22:52:19 dist-test-slave-kr25 krb5kdc[21754](info): commencing operation
krb5kdc: starting...
W20260927 22:52:19.999423 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:20 dist-test-slave-kr25 krb5kdc[21754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:20.230795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:20.543829 21767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:20.544101 21767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:20.544220 21767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:20.551239 21767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:20.551345 21767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:20.551419 21767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:20.551519 21767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:20.551577 21767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:20.561210 21767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:20.562448 21767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:20.563714 21767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:20.574196 21772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:20.574247 21775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:20.574286 21773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:20.575317 21767 server_base.cc:1034] running on GCE node
I20260927 22:52:20.576020 21767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:20.577252 21767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:20.578436 21767 hybrid_clock.cc:648] HybridClock initialized: now 1790549540578391 us; error 39 us; skew 500 ppm
I20260927 22:52:20.578727 21767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:20 dist-test-slave-kr25 krb5kdc[21754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:20.581530 21767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:20.582887 21767 webserver.cc:460] Webserver started at http://127.0.0.1:35325/ using document root <none> and password file <none>
I20260927 22:52:20.583544 21767 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:20.583671 21767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:20.583948 21767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:20.586748 21767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "7012bebfc3d04366a90be29384fa1773"
format_stamp: "Formatted at 2026-09-27 22:52:20 on dist-test-slave-kr25"
server_key: "7956a64dbf85b93faddfbe0a5bde30c6"
server_key_iv: "9963c11eacfd349bcff70ec9d5e86996"
server_key_version: "encryptionkey@0"
I20260927 22:52:20.587507 21767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "7012bebfc3d04366a90be29384fa1773"
format_stamp: "Formatted at 2026-09-27 22:52:20 on dist-test-slave-kr25"
server_key: "7956a64dbf85b93faddfbe0a5bde30c6"
server_key_iv: "9963c11eacfd349bcff70ec9d5e86996"
server_key_version: "encryptionkey@0"
I20260927 22:52:20.592249 21767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:52:20.595377 21782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:20.596668 21767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:52:20.596819 21767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "7012bebfc3d04366a90be29384fa1773"
format_stamp: "Formatted at 2026-09-27 22:52:20 on dist-test-slave-kr25"
server_key: "7956a64dbf85b93faddfbe0a5bde30c6"
server_key_iv: "9963c11eacfd349bcff70ec9d5e86996"
server_key_version: "encryptionkey@0"
I20260927 22:52:20.596982 21767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:20.614383 21767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:20.618021 21767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:20.618275 21767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:20.634532 21767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33383
I20260927 22:52:20.634560 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33383 every 8 connection(s)
I20260927 22:52:20.636077 21767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:20.639536 21834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:20.642314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21767
I20260927 22:52:20.642493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:20.642988 388 external_mini_cluster.cc:1428] Setting key 537c8c6795af931587f5942071f41aec
I20260927 22:52:20.650941 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773: Bootstrap starting.
I20260927 22:52:20.654480 21834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:20.655714 21834 log.cc:824] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:52:20 dist-test-slave-kr25 krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:52:20.658365 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773: No bootstrap required, opened a new log
I20260927 22:52:20.665925 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:20.644744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:36369 (local address 127.0.0.1:33383)
0927 22:52:20.645449 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:20.645460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:20.645499 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:52:20.647098 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:20.647125 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:20.647563 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:20.647875 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:20.649490 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.650738 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.652472 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.653106 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.653239 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.657827 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:20.657834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:20.657848 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:20.657880 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:20.660818 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:20.661555 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:20.661561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:20.661566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:20.661730 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:20.662158 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:20.662164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:20.662166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:20.662863 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:20.663194 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:20.663988 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 22:52:20.664455 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":322,"threads_started":1}
I20260927 22:52:20.667394 21834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7012bebfc3d04366a90be29384fa1773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33383 } }
I20260927 22:52:20.667868 21834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:20.667979 21834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7012bebfc3d04366a90be29384fa1773, State: Initialized, Role: FOLLOWER
I20260927 22:52:20.668640 21834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7012bebfc3d04366a90be29384fa1773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33383 } }
I20260927 22:52:20.668929 21834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:20.669059 21834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:20.669221 21834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:20.670522 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:20.672186 21834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7012bebfc3d04366a90be29384fa1773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33383 } }
I20260927 22:52:20.672602 21834 leader_election.cc:302] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7012bebfc3d04366a90be29384fa1773; no voters:
I20260927 22:52:20.673170 21834 leader_election.cc:288] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:20.673278 21839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:20.677109 21839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [term 1 LEADER]: Becoming Leader. State: Replica: 7012bebfc3d04366a90be29384fa1773, State: Running, Role: LEADER
I20260927 22:52:20.677706 21839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7012bebfc3d04366a90be29384fa1773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33383 } }
I20260927 22:52:20.679708 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7012bebfc3d04366a90be29384fa1773. Latest consensus state: current_term: 1 leader_uuid: "7012bebfc3d04366a90be29384fa1773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7012bebfc3d04366a90be29384fa1773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33383 } } }
I20260927 22:52:20.680052 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:20.681331 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7012bebfc3d04366a90be29384fa1773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7012bebfc3d04366a90be29384fa1773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33383 } } }
I20260927 22:52:20.681618 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:20.682193 21834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:20.686839 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:20.692147 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:20.675729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:50773 (local address 127.0.0.1:33383)
0927 22:52:20.675896 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:20.675903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:20.676409 (+ 506us) server_negotiation.cc:407] Connection header received
0927 22:52:20.676565 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:20.676570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:20.676649 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:20.676727 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:20.678135 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.678897 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.681811 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.682343 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.682425 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.685722 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:20.685727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:20.685730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:20.685761 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:20.688677 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:20.689542 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:20.689548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:20.689552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:20.689696 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:20.690257 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:20.690261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:20.690264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:20.690699 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:20.690965 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:20.691607 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 22:52:20.691794 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:52:20.692549 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:20.702622 21848 catalog_manager.cc:1347] Generated new cluster ID: a1702aae5aa440cf95bc3d856eee964a
I20260927 22:52:20.702754 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:20.728060 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:20.729322 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:20.734126 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7012bebfc3d04366a90be29384fa1773: Generated new TSK 0
I20260927 22:52:20.734855 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:20.740582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21767
I20260927 22:52:20.741976 21829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:20.830605 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21767
W20260927 22:52:20.844172 21752 connection.cc:535] client connection to 127.0.0.1:33383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33383): Connection reset by peer (error 104)
2026-09-27T22:52:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:20 dist-test-slave-kr25 krb5kdc[21866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:20 dist-test-slave-kr25 krb5kdc[21866](info): set up 2 sockets
Sep 27 22:52:20 dist-test-slave-kr25 krb5kdc[21866](info): commencing operation
krb5kdc: starting...
W20260927 22:52:21.843576 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:21 dist-test-slave-kr25 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:22.075117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:22.387682 21879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:22.387960 21879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:22.388048 21879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:22.395100 21879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:22.395198 21879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:22.395249 21879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:22.395306 21879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:22.395349 21879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:22.404837 21879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:22.406023 21879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:22.407238 21879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:22.418464 21884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:22.418437 21885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:22.418390 21887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:22.419227 21879 server_base.cc:1034] running on GCE node
I20260927 22:52:22.420046 21879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:22.421446 21879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:22.422669 21879 hybrid_clock.cc:648] HybridClock initialized: now 1790549542422616 us; error 47 us; skew 500 ppm
I20260927 22:52:22.422976 21879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:22 dist-test-slave-kr25 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:22.426003 21879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:22.427565 21879 webserver.cc:460] Webserver started at http://127.0.0.1:40199/ using document root <none> and password file <none>
I20260927 22:52:22.428262 21879 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:22.428360 21879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:22.428607 21879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:22.431598 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "30d5a2d270804eb68e8ed2bc5b53013b"
format_stamp: "Formatted at 2026-09-27 22:52:22 on dist-test-slave-kr25"
server_key: "c4da6712c0877b1c568cd2c6e7ba71f8"
server_key_iv: "6a02e1a3077ce9f9da8c53cd32ef456e"
server_key_version: "encryptionkey@0"
I20260927 22:52:22.432304 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "30d5a2d270804eb68e8ed2bc5b53013b"
format_stamp: "Formatted at 2026-09-27 22:52:22 on dist-test-slave-kr25"
server_key: "c4da6712c0877b1c568cd2c6e7ba71f8"
server_key_iv: "6a02e1a3077ce9f9da8c53cd32ef456e"
server_key_version: "encryptionkey@0"
I20260927 22:52:22.437186 21879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:52:22.440387 21894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:22.441610 21879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:52:22.441758 21879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "30d5a2d270804eb68e8ed2bc5b53013b"
format_stamp: "Formatted at 2026-09-27 22:52:22 on dist-test-slave-kr25"
server_key: "c4da6712c0877b1c568cd2c6e7ba71f8"
server_key_iv: "6a02e1a3077ce9f9da8c53cd32ef456e"
server_key_version: "encryptionkey@0"
I20260927 22:52:22.441911 21879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:22.459986 21879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:22.464007 21879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:22.464237 21879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:22.481539 21879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39799
I20260927 22:52:22.481565 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39799 every 8 connection(s)
I20260927 22:52:22.482911 21879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:22.486342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21879
I20260927 22:52:22.486506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:22.486788 21946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:22.486987 388 external_mini_cluster.cc:1428] Setting key eef04d38eaad51367ca6f8eccd905bd2
I20260927 22:52:22.500464 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:22.488754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:56869 (local address 127.0.0.1:39799)
0927 22:52:22.489540 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:22.489550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:22.489592 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:52:22.491543 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:22.491564 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:22.491970 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:22.492294 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:22.492901 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.494234 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.496066 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.496723 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.496861 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.498220 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:22.498228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:22.498242 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:22.498793 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:22.498869 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:22.498893 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:22.499004 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:22.499286 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 22:52:22.499499 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":334,"threads_started":1}
I20260927 22:52:22.500736 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b: Bootstrap starting.
I20260927 22:52:22.504268 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:22.504415 21946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:22.505463 21946 log.cc:824] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:22.508646 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b: No bootstrap required, opened a new log
I20260927 22:52:22.515214 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:22.508583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:43091 (local address 127.0.0.1:39799)
0927 22:52:22.508754 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:22.508760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:22.509217 (+ 457us) server_negotiation.cc:407] Connection header received
0927 22:52:22.509349 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:22.509355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:22.509439 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:22.509561 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:22.510013 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.511168 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.512494 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.513232 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.513363 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.514339 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:22.514346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:22.514351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:22.514376 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:22.514440 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:22.514468 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:22.514607 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:22.514759 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 22:52:22.514821 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:52:22.518167 21946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260927 22:52:22.518522 21946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:22.518594 21946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d5a2d270804eb68e8ed2bc5b53013b, State: Initialized, Role: FOLLOWER
I20260927 22:52:22.519083 21946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260927 22:52:22.519272 21946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:22.519362 21946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:22.519521 21946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:22.521586 21946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260927 22:52:22.522058 21946 leader_election.cc:302] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d5a2d270804eb68e8ed2bc5b53013b; no voters:
I20260927 22:52:22.522701 21946 leader_election.cc:288] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:22.522814 21956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:22.523381 21956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [term 1 LEADER]: Becoming Leader. State: Replica: 30d5a2d270804eb68e8ed2bc5b53013b, State: Running, Role: LEADER
I20260927 22:52:22.523916 21956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260927 22:52:22.524884 21946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:22.526615 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d5a2d270804eb68e8ed2bc5b53013b. Latest consensus state: current_term: 1 leader_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } } }
I20260927 22:52:22.526738 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d5a2d270804eb68e8ed2bc5b53013b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } } }
I20260927 22:52:22.526999 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:22.527099 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:22.529124 21963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:22.534440 21963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:22.542765 21963 catalog_manager.cc:1347] Generated new cluster ID: b93f5b7ca8c5494b9a90e9d3e45d13fe
I20260927 22:52:22.542878 21963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:22.556427 21963 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:22.557750 21963 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:22.564419 21963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d5a2d270804eb68e8ed2bc5b53013b: Generated new TSK 0
I20260927 22:52:22.565182 21963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:22.577998 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21879
W20260927 22:52:22.579355 21941 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:56869
W20260927 22:52:22.580101 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21879: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:22.580245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21879
W20260927 22:52:22.593493 21864 connection.cc:535] client connection to 127.0.0.1:39799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39799): Connection reset by peer (error 104)
2026-09-27T22:52:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1749 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:22 dist-test-slave-kr25 krb5kdc[21977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:22 dist-test-slave-kr25 krb5kdc[21977](info): set up 2 sockets
Sep 27 22:52:22 dist-test-slave-kr25 krb5kdc[21977](info): commencing operation
krb5kdc: starting...
W20260927 22:52:23.591527 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:23 dist-test-slave-kr25 krb5kdc[21977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:52:23.822798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:24.135092 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:24.135354 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:24.135496 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:24.142396 21990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:24.142498 21990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:24.142567 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:24.142647 21990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:24.142707 21990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:24.152172 21990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:24.153432 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:24.154697 21990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:24.165683 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:24.165795 21998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:24.165685 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:24.166405 21990 server_base.cc:1034] running on GCE node
I20260927 22:52:24.167177 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:24.168543 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:24.169740 21990 hybrid_clock.cc:648] HybridClock initialized: now 1790549544169702 us; error 32 us; skew 500 ppm
I20260927 22:52:24.170034 21990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:24 dist-test-slave-kr25 krb5kdc[21977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:24.176213 21990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:52:24.177579 21990 webserver.cc:460] Webserver started at http://127.0.0.1:32847/ using document root <none> and password file <none>
I20260927 22:52:24.178213 21990 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:24.178308 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:24.178544 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:24.181380 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "089d306a9a51438ab9014c3e6f478c72"
format_stamp: "Formatted at 2026-09-27 22:52:24 on dist-test-slave-kr25"
server_key: "fd93341baeb3d806f5edaacffc60e9f5"
server_key_iv: "9f3d7861a9a84bc241003063ad66a3eb"
server_key_version: "encryptionkey@0"
I20260927 22:52:24.182053 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "089d306a9a51438ab9014c3e6f478c72"
format_stamp: "Formatted at 2026-09-27 22:52:24 on dist-test-slave-kr25"
server_key: "fd93341baeb3d806f5edaacffc60e9f5"
server_key_iv: "9f3d7861a9a84bc241003063ad66a3eb"
server_key_version: "encryptionkey@0"
I20260927 22:52:24.186928 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:52:24.190236 22005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:24.191464 21990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:24.191629 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "089d306a9a51438ab9014c3e6f478c72"
format_stamp: "Formatted at 2026-09-27 22:52:24 on dist-test-slave-kr25"
server_key: "fd93341baeb3d806f5edaacffc60e9f5"
server_key_iv: "9f3d7861a9a84bc241003063ad66a3eb"
server_key_version: "encryptionkey@0"
I20260927 22:52:24.191799 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:24.197086 21990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:24.200704 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:24.200956 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:24.217283 21990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38249
I20260927 22:52:24.217324 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38249 every 8 connection(s)
I20260927 22:52:24.218590 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:24.222087 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:24.226912 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 21990
I20260927 22:52:24.227099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:24.227660 388 external_mini_cluster.cc:1428] Setting key d7b91e318499f22cdfc780e5d64ac3df
I20260927 22:52:24.233000 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72: Bootstrap starting.
I20260927 22:52:24.235170 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:24.229476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:38681 (local address 127.0.0.1:38249)
0927 22:52:24.230222 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:24.230234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:24.230272 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:52:24.231980 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:24.232003 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:24.232349 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:24.232685 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:24.233639 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:24.233645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:24.233658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:24.233672 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:24.233725 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:24.233743 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:24.233931 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:24.234053 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 22:52:24.234248 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":286,"threads_started":1}
I20260927 22:52:24.236153 22057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:24.237124 22057 log.cc:824] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:24.238521 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:24.240067 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72: No bootstrap required, opened a new log
I20260927 22:52:24.245884 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:24.243324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:55759 (local address 127.0.0.1:38249)
0927 22:52:24.243523 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:24.243530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:24.244187 (+ 657us) server_negotiation.cc:407] Connection header received
0927 22:52:24.244405 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:24.244410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:24.244463 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:24.244556 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:24.245306 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:24.245311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:24.245316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:24.245333 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:24.245378 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:24.245400 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:24.245509 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:24.245559 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 22:52:24.245613 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:52:24.248754 22057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d306a9a51438ab9014c3e6f478c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260927 22:52:24.249060 22057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:24.249130 22057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089d306a9a51438ab9014c3e6f478c72, State: Initialized, Role: FOLLOWER
I20260927 22:52:24.249593 22057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d306a9a51438ab9014c3e6f478c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260927 22:52:24.249801 22057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:24.249888 22057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:24.250000 22057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:24.252061 22057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d306a9a51438ab9014c3e6f478c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260927 22:52:24.252511 22057 leader_election.cc:302] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089d306a9a51438ab9014c3e6f478c72; no voters:
I20260927 22:52:24.253093 22057 leader_election.cc:288] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:24.253207 22067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:24.253830 22067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [term 1 LEADER]: Becoming Leader. State: Replica: 089d306a9a51438ab9014c3e6f478c72, State: Running, Role: LEADER
I20260927 22:52:24.254297 22067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d306a9a51438ab9014c3e6f478c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260927 22:52:24.254958 22057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:24.256815 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089d306a9a51438ab9014c3e6f478c72. Latest consensus state: current_term: 1 leader_uuid: "089d306a9a51438ab9014c3e6f478c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d306a9a51438ab9014c3e6f478c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } } }
I20260927 22:52:24.257151 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:24.257397 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "089d306a9a51438ab9014c3e6f478c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d306a9a51438ab9014c3e6f478c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } } }
I20260927 22:52:24.257699 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:24.259131 22072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:24.264295 22072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:24.272428 22072 catalog_manager.cc:1347] Generated new cluster ID: 46ddfe15789341139b6afbe892b1a703
I20260927 22:52:24.272550 22072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:24.284762 22072 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:24.285145 22072 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:24.300904 22072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089d306a9a51438ab9014c3e6f478c72: Generated new TSK 0
I20260927 22:52:24.301674 22072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:24.310653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21990
W20260927 22:52:24.312099 22052 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:38681
W20260927 22:52:24.312750 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 21990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:24.312867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 21990
2026-09-27T22:52:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1732 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:24 dist-test-slave-kr25 krb5kdc[22088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:24 dist-test-slave-kr25 krb5kdc[22088](info): set up 2 sockets
Sep 27 22:52:24 dist-test-slave-kr25 krb5kdc[22088](info): commencing operation
krb5kdc: starting...
W20260927 22:52:25.315799 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:25 dist-test-slave-kr25 krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:25.546433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43005
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:25.854816 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:25.855079 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:25.855175 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:25.862130 22101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:25.862232 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:25.862308 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:25.862380 22101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:25.862440 22101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:25.872341 22101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:25.873523 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:25.874751 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:25.885360 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:25.885334 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:25.885303 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:25.886101 22101 server_base.cc:1034] running on GCE node
I20260927 22:52:25.886790 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:25.888037 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:25.889236 22101 hybrid_clock.cc:648] HybridClock initialized: now 1790549545889199 us; error 30 us; skew 500 ppm
I20260927 22:52:25.889539 22101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:25 dist-test-slave-kr25 krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:25.892310 22101 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:25.893640 22101 webserver.cc:460] Webserver started at http://127.0.97.62:44109/ using document root <none> and password file <none>
I20260927 22:52:25.894269 22101 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:25.894380 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:25.894647 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:25.897464 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "47a1efa02b18442caa7961fcafed59c4"
format_stamp: "Formatted at 2026-09-27 22:52:25 on dist-test-slave-kr25"
server_key: "4465a2c59c9517e05ef987b1e1018951"
server_key_iv: "884325c8dacbca0844ccd93f64e6d44f"
server_key_version: "encryptionkey@0"
I20260927 22:52:25.898173 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "47a1efa02b18442caa7961fcafed59c4"
format_stamp: "Formatted at 2026-09-27 22:52:25 on dist-test-slave-kr25"
server_key: "4465a2c59c9517e05ef987b1e1018951"
server_key_iv: "884325c8dacbca0844ccd93f64e6d44f"
server_key_version: "encryptionkey@0"
I20260927 22:52:25.902854 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:52:25.906030 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:25.907292 22101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:52:25.907490 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "47a1efa02b18442caa7961fcafed59c4"
format_stamp: "Formatted at 2026-09-27 22:52:25 on dist-test-slave-kr25"
server_key: "4465a2c59c9517e05ef987b1e1018951"
server_key_iv: "884325c8dacbca0844ccd93f64e6d44f"
server_key_version: "encryptionkey@0"
I20260927 22:52:25.907666 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:25.926154 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:25.930013 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:25.930271 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:25.946748 22101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43005
I20260927 22:52:25.946765 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43005 every 8 connection(s)
I20260927 22:52:25.948160 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:25.951658 22168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:25.957716 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22101
I20260927 22:52:25.957856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:25.958254 388 external_mini_cluster.cc:1428] Setting key 6e4f88efb6bf3dca74d3ad9bcb2ba37b
I20260927 22:52:25.963914 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4: Bootstrap starting.
I20260927 22:52:25.966917 22168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:25.968132 22168 log.cc:824] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:25.970925 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4: No bootstrap required, opened a new log
Sep 27 22:52:25 dist-test-slave-kr25 krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:25.979236 22168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1efa02b18442caa7961fcafed59c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43005 } }
I20260927 22:52:25.979628 22168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:25.979738 22168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a1efa02b18442caa7961fcafed59c4, State: Initialized, Role: FOLLOWER
I20260927 22:52:25.980362 22168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1efa02b18442caa7961fcafed59c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43005 } }
I20260927 22:52:25.980588 22168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:25.980710 22168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:25.980882 22168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:25.983482 22168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1efa02b18442caa7961fcafed59c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43005 } }
I20260927 22:52:25.983973 22168 leader_election.cc:302] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47a1efa02b18442caa7961fcafed59c4; no voters:
I20260927 22:52:25.984586 22168 leader_election.cc:288] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:25.984671 22173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:25.986784 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:25.960093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36409 (local address 127.0.97.62:43005)
0927 22:52:25.963915 (+ 3822us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:25.963930 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:25.963977 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:52:25.966268 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:25.966296 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:25.966790 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:25.967201 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:25.970515 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.972435 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.973698 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.974729 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.974888 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:25.978997 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:25.979004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:25.979018 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:25.979056 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:25.982081 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:25.982936 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:25.982942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:25.982948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:25.983118 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:25.983598 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:25.983602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:25.983604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:25.984134 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:25.984398 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:25.984986 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 22:52:25.985353 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3591,"thread_start_us":3348,"threads_started":1}
I20260927 22:52:25.987947 22173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [term 1 LEADER]: Becoming Leader. State: Replica: 47a1efa02b18442caa7961fcafed59c4, State: Running, Role: LEADER
I20260927 22:52:25.988504 22173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1efa02b18442caa7961fcafed59c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43005 } }
I20260927 22:52:25.989014 22168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:25.990962 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:25.991153 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a1efa02b18442caa7961fcafed59c4. Latest consensus state: current_term: 1 leader_uuid: "47a1efa02b18442caa7961fcafed59c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1efa02b18442caa7961fcafed59c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43005 } } }
I20260927 22:52:25.991539 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:25.991989 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47a1efa02b18442caa7961fcafed59c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1efa02b18442caa7961fcafed59c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43005 } } }
I20260927 22:52:25.992237 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:25.995253 22178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:26.000393 22178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:26.019263 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:25.998243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50507 (local address 127.0.97.62:43005)
0927 22:52:25.998424 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:25.998431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:26.001902 (+ 3471us) server_negotiation.cc:407] Connection header received
0927 22:52:26.002063 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:26.002068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:26.002149 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:26.002246 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:26.004380 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.005638 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.007577 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.008300 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.008393 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:26.012029 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:26.012036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:26.012044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:26.012083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:26.015985 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:26.016882 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:26.016888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:26.016893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:26.017042 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:26.017517 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:26.017522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:26.017525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:26.017965 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:26.018253 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:26.018687 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 22:52:26.018880 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:52:26.024180 22178 catalog_manager.cc:1347] Generated new cluster ID: 51a61484d5004e8685e18ab50e1c0b5e
I20260927 22:52:26.024290 22178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:26.047564 22178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:26.048817 22178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:26.057152 22178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47a1efa02b18442caa7961fcafed59c4: Generated new TSK 0
I20260927 22:52:26.058245 22178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:26.069447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22101
I20260927 22:52:26.070854 22163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:26.162554 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22101
W20260927 22:52:26.176186 22086 connection.cc:535] client connection to 127.0.97.62:43005 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:43005): Connection reset by peer (error 104)
2026-09-27T22:52:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1850 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:26 dist-test-slave-kr25 krb5kdc[22200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:26 dist-test-slave-kr25 krb5kdc[22200](info): set up 2 sockets
Sep 27 22:52:26 dist-test-slave-kr25 krb5kdc[22200](info): commencing operation
krb5kdc: starting...
W20260927 22:52:27.179467 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:27 dist-test-slave-kr25 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:27.403991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38751
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:27.716472 22213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:27.716737 22213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:27.716866 22213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:27.723762 22213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:27.723857 22213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:27.723927 22213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:27.724009 22213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:27.724062 22213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:27.733479 22213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:27.734715 22213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:27.735957 22213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:27.746840 22218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:27.746953 22219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:27.746917 22221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:27.747921 22213 server_base.cc:1034] running on GCE node
I20260927 22:52:27.748663 22213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:27.749977 22213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:27.751169 22213 hybrid_clock.cc:648] HybridClock initialized: now 1790549547751126 us; error 41 us; skew 500 ppm
I20260927 22:52:27.751524 22213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:27 dist-test-slave-kr25 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:27.754459 22213 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:27.755904 22213 webserver.cc:460] Webserver started at http://127.0.97.62:35447/ using document root <none> and password file <none>
I20260927 22:52:27.756546 22213 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:27.756657 22213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:27.756927 22213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:27.759811 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "dbf30d58e8b24923bbb54d1a2b8b6082"
format_stamp: "Formatted at 2026-09-27 22:52:27 on dist-test-slave-kr25"
server_key: "f89d20c54d6c8b0ee71c5a7a187c7988"
server_key_iv: "dc73ab41f597de3d631dcca99bf6ce0f"
server_key_version: "encryptionkey@0"
I20260927 22:52:27.760515 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "dbf30d58e8b24923bbb54d1a2b8b6082"
format_stamp: "Formatted at 2026-09-27 22:52:27 on dist-test-slave-kr25"
server_key: "f89d20c54d6c8b0ee71c5a7a187c7988"
server_key_iv: "dc73ab41f597de3d631dcca99bf6ce0f"
server_key_version: "encryptionkey@0"
I20260927 22:52:27.765281 22213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:52:27.768388 22228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:27.769572 22213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:52:27.769711 22213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "dbf30d58e8b24923bbb54d1a2b8b6082"
format_stamp: "Formatted at 2026-09-27 22:52:27 on dist-test-slave-kr25"
server_key: "f89d20c54d6c8b0ee71c5a7a187c7988"
server_key_iv: "dc73ab41f597de3d631dcca99bf6ce0f"
server_key_version: "encryptionkey@0"
I20260927 22:52:27.769869 22213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:27.790383 22213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:27.794477 22213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:27.794720 22213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:27.811239 22213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38751
I20260927 22:52:27.811269 22279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38751 every 8 connection(s)
I20260927 22:52:27.812515 22213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:27.814910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22213
I20260927 22:52:27.815078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:27.815639 388 external_mini_cluster.cc:1428] Setting key d2b70aef6746a124cd367050325653a2
I20260927 22:52:27.816409 22280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:27.828074 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:27.817289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38023 (local address 127.0.97.62:38751)
0927 22:52:27.817994 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:27.818005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:27.818043 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:52:27.819581 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:27.819601 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:27.819971 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:27.820292 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:27.820815 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.822122 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.823853 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.824482 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.824609 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.825837 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:27.825844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:27.825861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:27.826393 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:27.826464 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:27.826491 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:27.826604 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:27.826878 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:52:27.827083 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":331,"threads_started":1}
I20260927 22:52:27.829241 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082: Bootstrap starting.
I20260927 22:52:27.831905 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:27.832767 22280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:27.833918 22280 log.cc:824] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:27.836670 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082: No bootstrap required, opened a new log
I20260927 22:52:27.843405 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:27.836860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54779 (local address 127.0.97.62:38751)
0927 22:52:27.837061 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:27.837068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:27.837430 (+ 362us) server_negotiation.cc:407] Connection header received
0927 22:52:27.837615 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:27.837618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:27.837674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:27.837750 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:27.838305 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.839079 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.840946 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.841508 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.841614 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.842736 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:27.842741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:27.842744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:27.842759 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:27.842795 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:27.842812 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:27.842894 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:27.843054 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 22:52:27.843095 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:52:27.846016 22280 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38751 } }
I20260927 22:52:27.846385 22280 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:27.846481 22280 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf30d58e8b24923bbb54d1a2b8b6082, State: Initialized, Role: FOLLOWER
I20260927 22:52:27.847123 22280 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38751 } }
I20260927 22:52:27.847327 22280 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:27.847419 22280 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:27.847577 22280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:27.849561 22280 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38751 } }
I20260927 22:52:27.849973 22280 leader_election.cc:302] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbf30d58e8b24923bbb54d1a2b8b6082; no voters:
I20260927 22:52:27.850474 22280 leader_election.cc:288] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:27.850610 22290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:27.851245 22290 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [term 1 LEADER]: Becoming Leader. State: Replica: dbf30d58e8b24923bbb54d1a2b8b6082, State: Running, Role: LEADER
I20260927 22:52:27.851781 22290 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38751 } }
I20260927 22:52:27.852392 22280 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:27.857928 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbf30d58e8b24923bbb54d1a2b8b6082. Latest consensus state: current_term: 1 leader_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38751 } } }
I20260927 22:52:27.858294 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:27.858744 22291 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf30d58e8b24923bbb54d1a2b8b6082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38751 } } }
I20260927 22:52:27.858992 22291 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082 [sys.catalog]: This master's current role is: LEADER
W20260927 22:52:27.865511 22304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:52:27.865631 22304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:52:27.865767 22305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:27.867300 22305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:27.875545 22305 catalog_manager.cc:1347] Generated new cluster ID: 5e128cd40b3e45a997cef40424af945c
I20260927 22:52:27.875653 22305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:27.888800 22305 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:27.890493 22305 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:27.911405 22305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbf30d58e8b24923bbb54d1a2b8b6082: Generated new TSK 0
I20260927 22:52:27.912276 22305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:27.922421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22213
W20260927 22:52:27.923941 22275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38023
W20260927 22:52:27.924693 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22213: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:27.924821 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22213
W20260927 22:52:27.938231 22198 connection.cc:535] client connection to 127.0.97.62:38751 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38751): Connection reset by peer (error 104)
2026-09-27T22:52:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1762 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:28 dist-test-slave-kr25 krb5kdc[22311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:28 dist-test-slave-kr25 krb5kdc[22311](info): set up 2 sockets
Sep 27 22:52:28 dist-test-slave-kr25 krb5kdc[22311](info): commencing operation
krb5kdc: starting...
W20260927 22:52:28.939469 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:29 dist-test-slave-kr25 krb5kdc[22311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:29.172477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35741
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:29.484723 22324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:29.485014 22324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:29.485114 22324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:29.492215 22324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:29.492318 22324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:29.492386 22324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:29.492475 22324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:29.492532 22324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:29.502899 22324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:29.504160 22324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:29.505396 22324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:29.516352 22329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:29.517023 22332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:29.517047 22330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:29.517570 22324 server_base.cc:1034] running on GCE node
I20260927 22:52:29.518321 22324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:29.519740 22324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:29.520942 22324 hybrid_clock.cc:648] HybridClock initialized: now 1790549549520888 us; error 47 us; skew 500 ppm
I20260927 22:52:29.521281 22324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:29 dist-test-slave-kr25 krb5kdc[22311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:29.527905 22324 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:29.529408 22324 webserver.cc:460] Webserver started at http://127.0.97.62:40781/ using document root <none> and password file <none>
I20260927 22:52:29.530084 22324 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:29.530196 22324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:29.530471 22324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:29.533406 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "25b3d3e73ebd41a9b197d71cca48cef0"
format_stamp: "Formatted at 2026-09-27 22:52:29 on dist-test-slave-kr25"
server_key: "afc2ff520fb926d093084ce560ea017c"
server_key_iv: "e7476d142cdce4f0e1de12da751861f4"
server_key_version: "encryptionkey@0"
I20260927 22:52:29.534158 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "25b3d3e73ebd41a9b197d71cca48cef0"
format_stamp: "Formatted at 2026-09-27 22:52:29 on dist-test-slave-kr25"
server_key: "afc2ff520fb926d093084ce560ea017c"
server_key_iv: "e7476d142cdce4f0e1de12da751861f4"
server_key_version: "encryptionkey@0"
I20260927 22:52:29.539117 22324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:52:29.542414 22339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:29.543748 22324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:52:29.543901 22324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "25b3d3e73ebd41a9b197d71cca48cef0"
format_stamp: "Formatted at 2026-09-27 22:52:29 on dist-test-slave-kr25"
server_key: "afc2ff520fb926d093084ce560ea017c"
server_key_iv: "e7476d142cdce4f0e1de12da751861f4"
server_key_version: "encryptionkey@0"
I20260927 22:52:29.544070 22324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:29.549501 22324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:29.553447 22324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:29.553706 22324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:29.570138 22324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35741
I20260927 22:52:29.570178 22390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35741 every 8 connection(s)
I20260927 22:52:29.571501 22324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:29.573792 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22324
I20260927 22:52:29.573941 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:29.574354 388 external_mini_cluster.cc:1428] Setting key 85e8d57825930cfab92266cf4ac02b56
I20260927 22:52:29.575207 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:29.582427 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:29.576046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33591 (local address 127.0.97.62:35741)
0927 22:52:29.576836 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:29.576851 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:29.576896 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:52:29.578575 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:29.578602 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:29.579037 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:29.579481 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:29.580415 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:29.580424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:29.580441 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:29.580457 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:29.580557 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:29.580583 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:29.580784 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:29.580913 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 22:52:29.581094 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":308,"threads_started":1}
I20260927 22:52:29.585794 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:29.589673 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0: Bootstrap starting.
I20260927 22:52:29.592958 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:29.590778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49965 (local address 127.0.97.62:35741)
0927 22:52:29.590953 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:29.590959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:29.591250 (+ 291us) server_negotiation.cc:407] Connection header received
0927 22:52:29.591375 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:29.591380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:29.591462 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:29.591582 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:29.592385 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:29.592390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:29.592395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:29.592412 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:29.592456 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:29.592477 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:29.592588 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:29.592636 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 22:52:29.592689 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:52:29.593276 22391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:29.594533 22391 log.cc:824] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:29.597179 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0: No bootstrap required, opened a new log
I20260927 22:52:29.604265 22391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35741 } }
I20260927 22:52:29.604609 22391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:29.604679 22391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b3d3e73ebd41a9b197d71cca48cef0, State: Initialized, Role: FOLLOWER
I20260927 22:52:29.605145 22391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35741 } }
I20260927 22:52:29.605321 22391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:29.605412 22391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:29.605533 22391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:29.607652 22391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35741 } }
I20260927 22:52:29.608084 22391 leader_election.cc:302] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25b3d3e73ebd41a9b197d71cca48cef0; no voters:
I20260927 22:52:29.608582 22391 leader_election.cc:288] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:29.608697 22401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:29.609267 22401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [term 1 LEADER]: Becoming Leader. State: Replica: 25b3d3e73ebd41a9b197d71cca48cef0, State: Running, Role: LEADER
I20260927 22:52:29.609792 22401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35741 } }
I20260927 22:52:29.610421 22391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:29.612372 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35741 } } }
I20260927 22:52:29.612764 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:29.612604 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25b3d3e73ebd41a9b197d71cca48cef0. Latest consensus state: current_term: 1 leader_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b3d3e73ebd41a9b197d71cca48cef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35741 } } }
I20260927 22:52:29.612883 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:29.614470 22410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:29.619472 22410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:29.628038 22410 catalog_manager.cc:1347] Generated new cluster ID: f6d6d39ec5034b7fb77f897c7c3581ad
I20260927 22:52:29.628165 22410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:29.637521 22410 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:29.638003 22410 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:29.644968 22410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25b3d3e73ebd41a9b197d71cca48cef0: Generated new TSK 0
I20260927 22:52:29.645963 22410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:29.654168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22324
W20260927 22:52:29.655268 22386 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33591
W20260927 22:52:29.655977 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22324: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:29.656107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22324
2026-09-27T22:52:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1730 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:29 dist-test-slave-kr25 krb5kdc[22422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:29 dist-test-slave-kr25 krb5kdc[22422](info): set up 2 sockets
Sep 27 22:52:29 dist-test-slave-kr25 krb5kdc[22422](info): commencing operation
krb5kdc: starting...
W20260927 22:52:30.659616 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:30 dist-test-slave-kr25 krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:30.902390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40461
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:31.211856 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:31.212122 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:31.212252 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:31.219148 22435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:31.219251 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:31.219329 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:31.219404 22435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:31.219481 22435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:31.228952 22435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:31.230203 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:31.231500 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:31.243037 22441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:31.243088 22443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:31.243032 22440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:31.243860 22435 server_base.cc:1034] running on GCE node
I20260927 22:52:31.244681 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:31.246055 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:31.247298 22435 hybrid_clock.cc:648] HybridClock initialized: now 1790549551247236 us; error 45 us; skew 500 ppm
I20260927 22:52:31.247701 22435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:31 dist-test-slave-kr25 krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:31.250548 22435 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:31.251957 22435 webserver.cc:460] Webserver started at http://127.0.97.62:46721/ using document root <none> and password file <none>
I20260927 22:52:31.252600 22435 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:31.252707 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:31.253029 22435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:31.255911 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "3c8a4c4ef45d4d458e946e42d972a35f"
format_stamp: "Formatted at 2026-09-27 22:52:31 on dist-test-slave-kr25"
server_key: "4da33d445efc6d1d0146da036c6aa206"
server_key_iv: "5dfdeef955c6d61d5406dc1054dbe6df"
server_key_version: "encryptionkey@0"
I20260927 22:52:31.256615 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c8a4c4ef45d4d458e946e42d972a35f"
format_stamp: "Formatted at 2026-09-27 22:52:31 on dist-test-slave-kr25"
server_key: "4da33d445efc6d1d0146da036c6aa206"
server_key_iv: "5dfdeef955c6d61d5406dc1054dbe6df"
server_key_version: "encryptionkey@0"
I20260927 22:52:31.261463 22435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:52:31.264741 22450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:31.265962 22435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:31.266108 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "3c8a4c4ef45d4d458e946e42d972a35f"
format_stamp: "Formatted at 2026-09-27 22:52:31 on dist-test-slave-kr25"
server_key: "4da33d445efc6d1d0146da036c6aa206"
server_key_iv: "5dfdeef955c6d61d5406dc1054dbe6df"
server_key_version: "encryptionkey@0"
I20260927 22:52:31.266271 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:31.277302 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:31.281065 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:31.281306 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:31.297816 22435 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40461
I20260927 22:52:31.297843 22501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40461 every 8 connection(s)
I20260927 22:52:31.299160 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:31.302696 22502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:31.303944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22435
I20260927 22:52:31.304090 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:31.304484 388 external_mini_cluster.cc:1428] Setting key 6789176e74d647372b6cf0294640882c
I20260927 22:52:31.316486 22502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f: Bootstrap starting.
Sep 27 22:52:31 dist-test-slave-kr25 krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:31.319623 22502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:31.320885 22502 log.cc:824] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:31.324097 22502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f: No bootstrap required, opened a new log
I20260927 22:52:31.328246 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:31.306220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58573 (local address 127.0.97.62:40461)
0927 22:52:31.306992 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:31.307007 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:31.307055 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:52:31.308905 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:31.308940 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:31.309481 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:31.309961 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:31.311245 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.313256 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.314564 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.315625 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.315794 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:31.319903 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:31.319915 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:31.319935 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:31.319993 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:31.323686 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:31.324473 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:31.324481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:31.324490 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:31.324699 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:31.325061 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:31.325074 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:31.325077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:31.325746 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:31.326053 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:31.326427 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 22:52:31.326881 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":318,"threads_started":1}
I20260927 22:52:31.332029 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:31.333345 22502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40461 } }
I20260927 22:52:31.333678 22502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:31.333770 22502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8a4c4ef45d4d458e946e42d972a35f, State: Initialized, Role: FOLLOWER
I20260927 22:52:31.334334 22502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40461 } }
I20260927 22:52:31.334578 22502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:31.334710 22502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:31.334867 22502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:31.337486 22502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40461 } }
I20260927 22:52:31.337935 22502 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8a4c4ef45d4d458e946e42d972a35f; no voters:
I20260927 22:52:31.338479 22502 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:31.338590 22512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:31.340394 22502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:31.345335 22512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [term 1 LEADER]: Becoming Leader. State: Replica: 3c8a4c4ef45d4d458e946e42d972a35f, State: Running, Role: LEADER
I20260927 22:52:31.345952 22512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40461 } }
I20260927 22:52:31.349006 22513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40461 } } }
I20260927 22:52:31.349153 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8a4c4ef45d4d458e946e42d972a35f. Latest consensus state: current_term: 1 leader_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a4c4ef45d4d458e946e42d972a35f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40461 } } }
I20260927 22:52:31.349589 22513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:31.349701 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:31.355194 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:31.336137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57187 (local address 127.0.97.62:40461)
0927 22:52:31.336312 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:31.336318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:31.336718 (+ 400us) server_negotiation.cc:407] Connection header received
0927 22:52:31.336857 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:31.336862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:31.336935 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:31.337030 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:31.338420 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.339562 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.341495 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.342205 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.342343 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:31.345489 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:31.345495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:31.345499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:31.345535 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:31.350892 (+ 5357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:31.351871 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:31.351876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:31.351880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:31.352068 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:31.352513 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:31.352519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:31.352522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:31.352972 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:31.354411 (+ 1439us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:31.354567 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:52:31.354798 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:52:31.358064 22521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:31.362534 22521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:31.370172 22521 catalog_manager.cc:1347] Generated new cluster ID: 564fca98d2e54883b4f32c7a76ee5610
I20260927 22:52:31.370301 22521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:31.389199 22521 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:31.390483 22521 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:31.400947 22521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8a4c4ef45d4d458e946e42d972a35f: Generated new TSK 0
I20260927 22:52:31.401754 22521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:31.406854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22435
I20260927 22:52:31.408175 22497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:31.496502 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22435
2026-09-27T22:52:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:31 dist-test-slave-kr25 krb5kdc[22534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:31 dist-test-slave-kr25 krb5kdc[22534](info): set up 2 sockets
Sep 27 22:52:31 dist-test-slave-kr25 krb5kdc[22534](info): commencing operation
krb5kdc: starting...
W20260927 22:52:32.499504 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:32 dist-test-slave-kr25 krb5kdc[22534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:32.732848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36009
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:33.041134 22547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:33.041446 22547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:33.041574 22547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:33.048659 22547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:33.048781 22547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:33.048859 22547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:33.048946 22547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:33.049002 22547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:33.058701 22547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:33.059983 22547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:33.061218 22547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:33.071906 22553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:33.072120 22555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:33.072360 22552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:33.073315 22547 server_base.cc:1034] running on GCE node
I20260927 22:52:33.074023 22547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:33.075366 22547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:33.076588 22547 hybrid_clock.cc:648] HybridClock initialized: now 1790549553076549 us; error 28 us; skew 500 ppm
I20260927 22:52:33.076906 22547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:33 dist-test-slave-kr25 krb5kdc[22534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:33.079849 22547 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:33.081176 22547 webserver.cc:460] Webserver started at http://127.0.97.62:44985/ using document root <none> and password file <none>
I20260927 22:52:33.081804 22547 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:33.081913 22547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:33.082217 22547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:33.085027 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "1d553a9a3f214ae2b7d2faa0315fb581"
format_stamp: "Formatted at 2026-09-27 22:52:33 on dist-test-slave-kr25"
server_key: "59582599e8b5f15c627816a5118efa70"
server_key_iv: "23ff4447efa85bafb3109b16a02ddc45"
server_key_version: "encryptionkey@0"
I20260927 22:52:33.085734 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "1d553a9a3f214ae2b7d2faa0315fb581"
format_stamp: "Formatted at 2026-09-27 22:52:33 on dist-test-slave-kr25"
server_key: "59582599e8b5f15c627816a5118efa70"
server_key_iv: "23ff4447efa85bafb3109b16a02ddc45"
server_key_version: "encryptionkey@0"
I20260927 22:52:33.090416 22547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:52:33.093652 22562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:33.094913 22547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:52:33.095083 22547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "1d553a9a3f214ae2b7d2faa0315fb581"
format_stamp: "Formatted at 2026-09-27 22:52:33 on dist-test-slave-kr25"
server_key: "59582599e8b5f15c627816a5118efa70"
server_key_iv: "23ff4447efa85bafb3109b16a02ddc45"
server_key_version: "encryptionkey@0"
I20260927 22:52:33.095257 22547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:33.133095 22547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:33.137003 22547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:33.137264 22547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:33.154268 22547 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36009
I20260927 22:52:33.154287 22613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36009 every 8 connection(s)
I20260927 22:52:33.155689 22547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:33.159332 22614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:33.163578 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22547
I20260927 22:52:33.163738 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:33.164145 388 external_mini_cluster.cc:1428] Setting key 73720fb3c29fdb7648523c8f3ba4d05a
I20260927 22:52:33.170218 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581: Bootstrap starting.
I20260927 22:52:33.173784 22614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:33.175076 22614 log.cc:824] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:33.177651 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:33.165933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45033 (local address 127.0.97.62:36009)
0927 22:52:33.166732 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:33.166747 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:33.166794 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:52:33.168783 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:33.168807 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:33.169221 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:33.169531 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:33.170104 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.171412 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.173245 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.173945 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.174085 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:33.175268 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:33.175276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:33.175291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:33.176017 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:33.176102 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:33.176143 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:33.176269 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:33.176449 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 22:52:33.176728 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":380,"threads_started":1}
I20260927 22:52:33.178067 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581: No bootstrap required, opened a new log
I20260927 22:52:33.181339 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:33.186228 22614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260927 22:52:33.186650 22614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:33.186753 22614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d553a9a3f214ae2b7d2faa0315fb581, State: Initialized, Role: FOLLOWER
I20260927 22:52:33.187398 22614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260927 22:52:33.187683 22614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:33.187808 22614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:33.187960 22614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:33.190577 22614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260927 22:52:33.191211 22614 leader_election.cc:302] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d553a9a3f214ae2b7d2faa0315fb581; no voters:
I20260927 22:52:33.191401 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:33.185239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37291 (local address 127.0.97.62:36009)
0927 22:52:33.185410 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:33.185415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:33.185731 (+ 316us) server_negotiation.cc:407] Connection header received
0927 22:52:33.185885 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:33.185889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:33.185965 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:33.186065 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:33.186451 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.187490 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.188856 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.189522 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.189662 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:33.190674 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:33.190680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:33.190684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:33.190706 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:33.190753 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:33.190773 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:33.190858 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:33.190979 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:52:33.191068 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:52:33.191962 22614 leader_election.cc:288] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:33.192097 22624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:33.192721 22624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [term 1 LEADER]: Becoming Leader. State: Replica: 1d553a9a3f214ae2b7d2faa0315fb581, State: Running, Role: LEADER
I20260927 22:52:33.193318 22624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260927 22:52:33.195024 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d553a9a3f214ae2b7d2faa0315fb581. Latest consensus state: current_term: 1 leader_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } }
I20260927 22:52:33.195286 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:33.195901 22626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d553a9a3f214ae2b7d2faa0315fb581" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } }
I20260927 22:52:33.196283 22626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:33.200096 22614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:33.200243 22628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:33.205839 22628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:33.214146 22628 catalog_manager.cc:1347] Generated new cluster ID: 63bfcf2f354243e9a2e2f8db2849ee1b
I20260927 22:52:33.214258 22628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:33.225317 22628 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:33.226629 22628 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:33.235553 22628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d553a9a3f214ae2b7d2faa0315fb581: Generated new TSK 0
I20260927 22:52:33.236670 22628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:33.252609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22547
W20260927 22:52:33.253769 22609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45033
W20260927 22:52:33.254508 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:33.254644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22547
2026-09-27T22:52:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:33 dist-test-slave-kr25 krb5kdc[22645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:33 dist-test-slave-kr25 krb5kdc[22645](info): set up 2 sockets
Sep 27 22:52:33 dist-test-slave-kr25 krb5kdc[22645](info): commencing operation
krb5kdc: starting...
W20260927 22:52:34.283497 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:34 dist-test-slave-kr25 krb5kdc[22645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:34.512939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33965
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:34.823951 22658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:34.824229 22658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:34.824365 22658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:34.831336 22658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:34.831462 22658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:34.831542 22658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:34.831625 22658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:34.831681 22658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:34.841209 22658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:34.842473 22658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:34.843791 22658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:34.854496 22663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:34.854455 22664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:34.854517 22666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:34.855127 22658 server_base.cc:1034] running on GCE node
I20260927 22:52:34.855898 22658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:34.857203 22658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:34.858397 22658 hybrid_clock.cc:648] HybridClock initialized: now 1790549554858354 us; error 37 us; skew 500 ppm
I20260927 22:52:34.858719 22658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:34 dist-test-slave-kr25 krb5kdc[22645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:34.864686 22658 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:34.866104 22658 webserver.cc:460] Webserver started at http://127.0.97.62:43751/ using document root <none> and password file <none>
I20260927 22:52:34.866778 22658 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:34.866895 22658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:34.867199 22658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:34.870236 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "a701b57d0da749a1b540c5176f5bf256"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-kr25"
server_key: "4766b32aacd59722c238274b6bab1d4b"
server_key_iv: "552bbb395ced3ca6110b14fea5c5720c"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.870994 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "a701b57d0da749a1b540c5176f5bf256"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-kr25"
server_key: "4766b32aacd59722c238274b6bab1d4b"
server_key_iv: "552bbb395ced3ca6110b14fea5c5720c"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.876283 22658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260927 22:52:34.879719 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:34.881031 22658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:52:34.881206 22658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "a701b57d0da749a1b540c5176f5bf256"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-kr25"
server_key: "4766b32aacd59722c238274b6bab1d4b"
server_key_iv: "552bbb395ced3ca6110b14fea5c5720c"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.881383 22658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:34.886740 22658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:34.890348 22658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:34.890596 22658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:34.906558 22658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33965
I20260927 22:52:34.906595 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33965 every 8 connection(s)
I20260927 22:52:34.907920 22658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:34.911556 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:34.913398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22658
I20260927 22:52:34.913543 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:34.913937 388 external_mini_cluster.cc:1428] Setting key 6d4c990086ffbd08e8120d6141813761
I20260927 22:52:34.921875 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:34.915676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34315 (local address 127.0.97.62:33965)
0927 22:52:34.916479 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:34.916494 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:34.916544 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:52:34.918212 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:34.918245 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:34.918698 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:34.919126 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:34.920028 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:34.920036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:34.920064 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:34.920082 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:34.920151 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:34.920183 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:34.920393 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:34.920514 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:52:34.920696 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":330,"threads_started":1}
I20260927 22:52:34.925369 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256: Bootstrap starting.
I20260927 22:52:34.925488 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:34.928287 22725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:34.929224 22725 log.cc:824] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:34.931897 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:34.929551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53503 (local address 127.0.97.62:33965)
0927 22:52:34.929734 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:34.929741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:34.930082 (+ 341us) server_negotiation.cc:407] Connection header received
0927 22:52:34.930249 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:34.930253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:34.930298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:34.930376 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:34.931270 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:34.931275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:34.931280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:34.931297 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:34.931341 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:34.931360 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:34.931490 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:34.931585 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:52:34.931639 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:52:34.932557 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256: No bootstrap required, opened a new log
I20260927 22:52:34.939847 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a701b57d0da749a1b540c5176f5bf256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33965 } }
I20260927 22:52:34.940181 22725 raft_consensus.cc:374] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:34.940251 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a701b57d0da749a1b540c5176f5bf256, State: Initialized, Role: FOLLOWER
I20260927 22:52:34.940828 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a701b57d0da749a1b540c5176f5bf256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33965 } }
I20260927 22:52:34.941133 22725 raft_consensus.cc:388] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:34.941280 22725 raft_consensus.cc:482] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:34.941449 22725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:34.943502 22725 raft_consensus.cc:504] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a701b57d0da749a1b540c5176f5bf256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33965 } }
I20260927 22:52:34.943940 22725 leader_election.cc:302] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a701b57d0da749a1b540c5176f5bf256; no voters:
I20260927 22:52:34.944442 22725 leader_election.cc:288] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:34.944571 22735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:34.945151 22735 raft_consensus.cc:686] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [term 1 LEADER]: Becoming Leader. State: Replica: a701b57d0da749a1b540c5176f5bf256, State: Running, Role: LEADER
I20260927 22:52:34.945663 22735 consensus_queue.cc:237] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a701b57d0da749a1b540c5176f5bf256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33965 } }
I20260927 22:52:34.946353 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:34.948136 22737 sys_catalog.cc:455] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a701b57d0da749a1b540c5176f5bf256. Latest consensus state: current_term: 1 leader_uuid: "a701b57d0da749a1b540c5176f5bf256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a701b57d0da749a1b540c5176f5bf256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33965 } } }
I20260927 22:52:34.948514 22737 sys_catalog.cc:458] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:34.948990 22736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a701b57d0da749a1b540c5176f5bf256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a701b57d0da749a1b540c5176f5bf256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33965 } } }
I20260927 22:52:34.949281 22736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:34.949529 22742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:34.955163 22742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:34.963255 22742 catalog_manager.cc:1347] Generated new cluster ID: fa298c9c981c4d398423e0970a20d9e3
I20260927 22:52:34.963377 22742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:34.987046 22742 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:34.987449 22742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:34.995261 22742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a701b57d0da749a1b540c5176f5bf256: Generated new TSK 0
I20260927 22:52:34.996069 22742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:35.005615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22658
W20260927 22:52:35.006618 22720 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34315
W20260927 22:52:35.007314 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:52:35.007467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22658
2026-09-27T22:52:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1752 ms)
[----------] 18 tests from AuthTokenIssuingTest (32091 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:35 dist-test-slave-kr25 krb5kdc[22756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:35 dist-test-slave-kr25 krb5kdc[22756](info): set up 2 sockets
Sep 27 22:52:35 dist-test-slave-kr25 krb5kdc[22756](info): commencing operation
krb5kdc: starting...
W20260927 22:52:36.015700 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:36 dist-test-slave-kr25 krb5kdc[22756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:36.255077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46409
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46409,127.0.97.61:42651,127.0.97.60:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:36.570511 22769 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:52:36.570829 22769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:36.570938 22769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:36.571051 22769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:36.578094 22769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:36.578186 22769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:36.578272 22769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:36.578338 22769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:36.578425 22769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:36.588351 22769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46409,127.0.97.61:42651,127.0.97.60:41431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:36.589628 22769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:36.590930 22769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:36.603868 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:36.603925 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:36.603896 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:36.604709 22769 server_base.cc:1034] running on GCE node
I20260927 22:52:36.605398 22769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:36.606678 22769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:36.607869 22769 hybrid_clock.cc:648] HybridClock initialized: now 1790549556607810 us; error 36 us; skew 500 ppm
I20260927 22:52:36.608161 22769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:36 dist-test-slave-kr25 krb5kdc[22756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:36.611054 22769 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:36.612435 22769 webserver.cc:460] Webserver started at http://127.0.97.62:40221/ using document root <none> and password file <none>
I20260927 22:52:36.613089 22769 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:36.613183 22769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:36.613483 22769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:36.616300 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "00236005e51440f098eb56e9310e6d6e"
format_stamp: "Formatted at 2026-09-27 22:52:36 on dist-test-slave-kr25"
server_key: "bd56750ef5d4d16f4be0220a8389e5f8"
server_key_iv: "5b7cbbb9bad06462601c381ecaaa1b28"
server_key_version: "encryptionkey@0"
I20260927 22:52:36.617038 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "00236005e51440f098eb56e9310e6d6e"
format_stamp: "Formatted at 2026-09-27 22:52:36 on dist-test-slave-kr25"
server_key: "bd56750ef5d4d16f4be0220a8389e5f8"
server_key_iv: "5b7cbbb9bad06462601c381ecaaa1b28"
server_key_version: "encryptionkey@0"
I20260927 22:52:36.621789 22769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:52:36.624992 22784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:36.626228 22769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:52:36.626379 22769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "00236005e51440f098eb56e9310e6d6e"
format_stamp: "Formatted at 2026-09-27 22:52:36 on dist-test-slave-kr25"
server_key: "bd56750ef5d4d16f4be0220a8389e5f8"
server_key_iv: "5b7cbbb9bad06462601c381ecaaa1b28"
server_key_version: "encryptionkey@0"
I20260927 22:52:36.626569 22769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:36.643899 22769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:36.647639 22769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:36.647864 22769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:36.664598 22769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46409
I20260927 22:52:36.664569 22835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46409 every 8 connection(s)
I20260927 22:52:36.666157 22769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:36.670290 22836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:36.675685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22769
I20260927 22:52:36.675895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:36.676648 388 external_mini_cluster.cc:1428] Setting key 977c5f24dffefb4561ca0820a9a3cfd2
I20260927 22:52:36.677095 22836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:52:36 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:36.699203 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.680454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46409 (local address 127.0.0.1:44758)
0927 22:52:36.681284 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.681335 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:36.683262 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:36.684213 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:36.684218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:36.684743 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:36.685091 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:36.685109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:36.686824 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:36.686827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:36.687916 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:36.687927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:36.688830 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:36.688836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:36.689067 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:36.689637 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:36.689658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:36.691833 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:36.695331 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:36.695340 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:36.695348 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:36.695763 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:36.696190 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:36.696193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:36.696195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:36.696333 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:36.697007 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:36.697015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:36.697219 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 22:52:36.697419 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 22:52:36.697964 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":135,"thread_start_us":320,"threads_started":1}
I20260927 22:52:36.699244 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.681694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.0.97.62:46409)
0927 22:52:36.682344 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:36.682351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:36.682400 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:52:36.683647 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:36.683654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:36.683971 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:36.684177 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:36.685298 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:36.686691 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:36.688052 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:36.688610 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:36.688788 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:36.692061 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:36.692071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:36.692085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:36.692121 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:36.695217 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:36.695895 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:36.695903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:36.695907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:36.696093 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:36.696439 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:36.696443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:36.696445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:36.696879 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:36.697080 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:36.697561 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 22:52:36.697853 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":441,"thread_start_us":288,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 22:52:36.702122 22836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 22:52:36.703159 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.702798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37664)
0927 22:52:36.702950 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.703048 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:52:36.703428 22788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
I20260927 22:52:36.704857 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.704536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37674)
0927 22:52:36.704674 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.704747 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:52:36.705027 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42651: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:52:36.728912 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42651
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46409,127.0.97.61:42651,127.0.97.60:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:52:36.754385 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } attempt: 1
I20260927 22:52:36.755612 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.755077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37684)
0927 22:52:36.755303 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.755412 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:52:36.756543 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.756192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37696)
0927 22:52:36.756328 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.756407 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:52:36.756711 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42651: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
I20260927 22:52:36.824960 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } attempt: 2
I20260927 22:52:36.826187 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.825639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37712)
0927 22:52:36.825892 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.826013 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:52:36.827231 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.826828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37714)
0927 22:52:36.826976 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.827083 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:52:36.827462 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42651: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
I20260927 22:52:36.918751 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } attempt: 3
I20260927 22:52:36.919961 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.919470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37716)
0927 22:52:36.919681 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.919789 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:52:36.920934 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:36.920546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37726)
0927 22:52:36.920700 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:36.920781 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:52:36.921150 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42651: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
W20260927 22:52:37.041800 22843 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:52:37.042129 22843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:37.042235 22843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:37.042347 22843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:37.049520 22843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:37.049610 22843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:37.049682 22843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:37.049751 22843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:37.049832 22843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:37.059842 22843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46409,127.0.97.61:42651,127.0.97.60:41431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:37.061160 22843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:37.062722 22843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:52:37.064394 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } attempt: 4
I20260927 22:52:37.065653 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.064994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37730)
0927 22:52:37.065386 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.065492 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260927 22:52:37.066641 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.066244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37734)
0927 22:52:37.066398 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.066492 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:52:37.066815 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42651: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42651: connect: Connection refused (error 111)
W20260927 22:52:37.074606 22849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:37.074730 22848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:37.074913 22851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:37.075719 22843 server_base.cc:1034] running on GCE node
I20260927 22:52:37.076467 22843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:37.077865 22843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:37.079084 22843 hybrid_clock.cc:648] HybridClock initialized: now 1790549557079014 us; error 54 us; skew 500 ppm
I20260927 22:52:37.079399 22843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:37.082329 22843 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:52:37.083662 22843 webserver.cc:460] Webserver started at http://127.0.97.61:35277/ using document root <none> and password file <none>
I20260927 22:52:37.084295 22843 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:37.084379 22843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:37.084712 22843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:37.087810 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data/instance:
uuid: "b5b0d427e07b4b61bd5a41810cbb7221"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-kr25"
server_key: "99e12a78920e4e1474ed13a59f0e51cd"
server_key_iv: "213f3f947e0b587b6f6d990c166ec211"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.088514 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/wal/instance:
uuid: "b5b0d427e07b4b61bd5a41810cbb7221"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-kr25"
server_key: "99e12a78920e4e1474ed13a59f0e51cd"
server_key_iv: "213f3f947e0b587b6f6d990c166ec211"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.093405 22843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260927 22:52:37.096732 22858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:37.098058 22843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:52:37.098214 22843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/wal
uuid: "b5b0d427e07b4b61bd5a41810cbb7221"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-kr25"
server_key: "99e12a78920e4e1474ed13a59f0e51cd"
server_key_iv: "213f3f947e0b587b6f6d990c166ec211"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.098402 22843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:37.117436 22843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:37.121622 22843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:37.121838 22843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:37.137454 22843 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42651
I20260927 22:52:37.137501 22909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42651 every 8 connection(s)
I20260927 22:52:37.138733 22843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:52:37.139828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22843
I20260927 22:52:37.139978 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-1/wal/instance
I20260927 22:52:37.140390 388 external_mini_cluster.cc:1428] Setting key b3cb0052b824643e5ec7398fb5247be7
I20260927 22:52:37.142465 22910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:37.149369 22910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:37.169937 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.152117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.0.97.62:46409)
0927 22:52:37.152283 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.152287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.153578 (+ 1291us) server_negotiation.cc:407] Connection header received
0927 22:52:37.155455 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.155461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.155540 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.155634 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.157126 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.157806 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.159365 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.159820 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.159932 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.164446 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.164452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.164455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.164490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.166923 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.167747 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.167751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.167753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.167875 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.168303 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.168306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.168308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.168591 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.168697 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.169525 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.169715 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:52:37.171262 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.152651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46409 (local address 127.0.0.1:44766)
0927 22:52:37.153409 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.153451 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.155166 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.155879 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.155887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.156390 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.156953 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.156971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.157952 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.157955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.159219 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.159231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.160033 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.160039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.160267 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.160873 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.160895 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.164205 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.167067 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.167075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.167092 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.167644 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.167968 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.167972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.167975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.168200 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.168701 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.168709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.169151 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.169433 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.169938 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":319,"threads_started":1}
I20260927 22:52:37.173134 22910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:52:37.186694 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.173884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37740)
0927 22:52:37.174079 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.174097 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.174197 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.175136 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.175141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.175586 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.175720 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.175728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.177040 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.177045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.177845 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.177855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.178641 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.178647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.178793 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.179315 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.179331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.181431 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.184424 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.184430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.184434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.184771 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.185111 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.185116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.185119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.185227 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.185785 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.185789 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.185922 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.186037 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.186315 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 22:52:37.186793 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.174082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.97.61:42651)
0927 22:52:37.174747 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.174753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.174778 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:52:37.174818 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.174824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.175003 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.175127 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.175866 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.176900 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.177978 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.178497 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.178604 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.181623 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.181633 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.181647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.181678 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.184311 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.184883 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.184889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.184892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.184998 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.185316 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.185321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.185324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.185656 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.185847 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.186182 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.186423 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":531,"thread_start_us":382,"threads_started":1}
I20260927 22:52:37.189154 22910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:37.190217 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.189783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45770)
0927 22:52:37.189936 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.190066 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:52:37.190460 22859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:52:37.191915 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.191586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45786)
0927 22:52:37.191721 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.191808 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:52:37.192098 22910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
I20260927 22:52:37.192677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41431
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46409,127.0.97.61:42651,127.0.97.60:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:52:37.241382 22910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 1
I20260927 22:52:37.242434 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.242008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45798)
0927 22:52:37.242200 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.242286 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:52:37.243476 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.243136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45806)
0927 22:52:37.243256 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.243342 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 22:52:37.243675 22910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
I20260927 22:52:37.311872 22910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 2
I20260927 22:52:37.312995 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.312552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45820)
0927 22:52:37.312748 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.312848 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:52:37.313927 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.313643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45828)
0927 22:52:37.313753 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.313817 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 22:52:37.314131 22910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
I20260927 22:52:37.366086 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } attempt: 5
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:52:37.380091 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.366961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.0.97.61:42651)
0927 22:52:37.367180 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.367186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.367206 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:52:37.367297 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.367302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.367369 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.367469 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.368406 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.369489 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.370654 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.371323 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.371478 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.374904 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.374909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.374912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.374945 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.377466 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.378314 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.378318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.378320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.378435 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.378835 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.378838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.378840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.379106 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.379211 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.379692 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.379859 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:52:37.380256 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.366849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37750)
0927 22:52:37.367030 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.367051 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.367178 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.367525 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.367530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.368011 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.368232 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.368245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.369666 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.369670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.370499 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.370511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.371487 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.371492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.371673 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.372234 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.372253 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.374699 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.377619 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.377624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.377628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.378205 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.378536 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.378551 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.378555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.378740 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.379219 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.379224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.379414 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.379613 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.379903 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:52:37.381409 22836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:37.382471 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.382042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45838)
0927 22:52:37.382219 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.382307 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:52:37.383374 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.383030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45850)
0927 22:52:37.383165 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.383259 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 22:52:37.383646 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
I20260927 22:52:37.405315 22910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 3
I20260927 22:52:37.406389 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.405949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45858)
0927 22:52:37.406150 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.406238 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:52:37.407370 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.406986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45862)
0927 22:52:37.407195 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.407266 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260927 22:52:37.407590 22910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
I20260927 22:52:37.434887 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 1
I20260927 22:52:37.436100 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.435579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45876)
0927 22:52:37.435786 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.435900 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:52:37.437114 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.436757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45886)
0927 22:52:37.436895 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.436983 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 22:52:37.437312 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
I20260927 22:52:37.505527 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 2
I20260927 22:52:37.506697 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.506176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45894)
0927 22:52:37.506393 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.506496 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:52:37.507658 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.507276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45910)
0927 22:52:37.507406 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.507515 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 22:52:37.507843 22836 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
W20260927 22:52:37.509661 22917 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:52:37.509977 22917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:37.510072 22917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:37.510181 22917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:37.517381 22917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:37.517460 22917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:37.517546 22917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:37.517611 22917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:37.517690 22917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:37.528379 22917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46409,127.0.97.61:42651,127.0.97.60:41431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:37.529624 22917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:37.530862 22917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:37.541615 22923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:37.541667 22922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:37.541821 22925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:37.542513 22917 server_base.cc:1034] running on GCE node
I20260927 22:52:37.543239 22917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:37.544623 22917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:37.545835 22917 hybrid_clock.cc:648] HybridClock initialized: now 1790549557545771 us; error 40 us; skew 500 ppm
I20260927 22:52:37.546132 22917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:37.549038 22917 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:52:37.550438 22917 webserver.cc:460] Webserver started at http://127.0.97.60:41111/ using document root <none> and password file <none>
I20260927 22:52:37.550772 22910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 4
I20260927 22:52:37.551131 22917 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:37.551251 22917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:37.551644 22917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:37.551864 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.551374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45926)
0927 22:52:37.551619 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.551706 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 22:52:37.552810 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.552527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45932)
0927 22:52:37.552642 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.552709 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 22:52:37.552992 22910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41431: connect: Connection refused (error 111)
I20260927 22:52:37.554807 22917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data/instance:
uuid: "470efaa4d8934b93a29ff19ab3afed70"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-kr25"
server_key: "9af3f15fad9dad976cf14ae63ff64839"
server_key_iv: "5bb33c4ce95d21d43bcd74aded9cfae9"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.555569 22917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/wal/instance:
uuid: "470efaa4d8934b93a29ff19ab3afed70"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-kr25"
server_key: "9af3f15fad9dad976cf14ae63ff64839"
server_key_iv: "5bb33c4ce95d21d43bcd74aded9cfae9"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.560388 22917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:52:37.563576 22932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:37.564829 22917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:37.564975 22917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/wal
uuid: "470efaa4d8934b93a29ff19ab3afed70"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-kr25"
server_key: "9af3f15fad9dad976cf14ae63ff64839"
server_key_iv: "5bb33c4ce95d21d43bcd74aded9cfae9"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.565160 22917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:37.583528 22917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:37.587226 22917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:37.587512 22917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:37.604305 22917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41431
I20260927 22:52:37.604329 22983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41431 every 8 connection(s)
I20260927 22:52:37.605575 22917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:52:37.609192 22984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:37.614035 22836 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 3
I20260927 22:52:37.614542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 22917
I20260927 22:52:37.614717 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549216186161-388-0/minicluster-data/master-2/wal/instance
I20260927 22:52:37.615332 388 external_mini_cluster.cc:1428] Setting key b0d9db7587b787bd46db60cc15dc6213
I20260927 22:52:37.615720 22984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:52:37.635488 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.618839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.97.62:46409)
0927 22:52:37.619043 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.619048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.619666 (+ 618us) server_negotiation.cc:407] Connection header received
0927 22:52:37.620431 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.620436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.620506 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.620611 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.622583 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.623583 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.625458 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.625982 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.626118 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.629403 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.629408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.629411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.629440 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.632235 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.633153 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.633158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.633162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.633289 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.633703 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.633708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.633712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.634005 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.634170 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.634872 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.635095 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:52:37.636554 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.618761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46409 (local address 127.0.0.1:44778)
0927 22:52:37.619527 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.619571 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.620181 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.621169 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.621176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.621942 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.622399 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.622420 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.623785 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.623791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.625337 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.625348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.626195 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.626202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.626365 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.626925 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.626947 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.629235 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.632355 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.632366 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.632381 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.633042 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.633393 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.633398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.633402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.633599 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.634155 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.634164 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.634496 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.634758 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.635219 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":27,"thread_start_us":358,"threads_started":1}
I20260927 22:52:37.637852 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.614708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45946)
0927 22:52:37.615018 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.615039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.615168 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.620654 (+ 5486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.620659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.621144 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.621324 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.621337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.623630 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.623635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.624998 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.625012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.625864 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.625870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.626067 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.626596 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.626614 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.630300 (+ 3686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.635074 (+ 4774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.635079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.635083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.635593 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.636063 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.636068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.636072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.636193 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.636921 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.636925 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.637118 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.637251 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.637505 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260927 22:52:37.638298 22984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:37.638545 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.616676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.97.60:41431)
0927 22:52:37.617247 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.617252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.617275 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:52:37.620172 (+ 2897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.620196 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.620458 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.621510 (+ 1052us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.622087 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.623372 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.624721 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.625589 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.625758 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.630682 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.630691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.630696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.630726 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.635024 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.635759 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.635767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.635771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.635928 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.636363 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.636367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.636370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.636773 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.637078 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.637705 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.638137 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":344,"server-negotiator.queue_time_us":453,"spinlock_wait_cycles":16384,"thread_start_us":301,"threads_started":1}
I20260927 22:52:37.638813 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.615736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.0.97.60:41431)
0927 22:52:37.616583 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.616600 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.616662 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:52:37.620171 (+ 3509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.620198 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.620460 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.620639 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.621512 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.623404 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.625146 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.625706 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.626349 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.630495 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.630503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.630515 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.630543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.634921 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.635759 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.635767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.635771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.635933 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.636295 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.636298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.636300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.636769 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.638045 (+ 1276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.638195 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.638392 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":609,"spinlock_wait_cycles":51712,"thread_start_us":372,"threads_started":1}
I20260927 22:52:37.645273 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e: Bootstrap starting.
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:52:37.649123 22836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:37.650286 22836 log.cc:824] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:37.653227 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.639206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.0.97.61:42651)
0927 22:52:37.639423 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.639429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.639839 (+ 410us) server_negotiation.cc:407] Connection header received
0927 22:52:37.639956 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.639962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.640043 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.640151 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.640832 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.641809 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.643415 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.643894 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.644010 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.647373 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.647379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.647382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.647418 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.650561 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.651302 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.651308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.651311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.651494 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.651932 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.651936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.651939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.652273 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.652363 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.652809 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.652975 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:52:37.653466 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e: No bootstrap required, opened a new log
I20260927 22:52:37.653901 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.639635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42651 (local address 127.0.0.1:37760)
0927 22:52:37.639768 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.639782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.639862 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.640167 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.640170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.640487 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.640664 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.640675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.641986 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.641992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.643269 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.643283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.644050 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.644056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.644240 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.644768 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.644788 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.647174 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.650680 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.650687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.650691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.651192 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.651636 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.651640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.651644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.651805 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.652435 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.652439 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.652556 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.653446 (+ 890us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.653683 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 22:52:37.654767 22984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:52:37.663062 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.646596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.0.97.62:46409)
0927 22:52:37.646767 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.646773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.646792 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:52:37.646854 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.646859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.646940 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.647070 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.648502 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.649624 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.651263 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.651954 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.652080 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.656974 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.656981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.656985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.657022 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.660273 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.660881 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.660887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.660891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.661053 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.661438 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.661444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.661448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.661818 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.662116 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.662456 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.662679 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:52:37.664158 22836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.664554 22836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00236005e51440f098eb56e9310e6d6e, State: Initialized, Role: FOLLOWER
I20260927 22:52:37.665094 22836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.666160 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.666680 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:37.667414 22836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:37.668735 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.655604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.0.97.60:41431)
0927 22:52:37.655761 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.655767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.655788 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:52:37.655830 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.655835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.655895 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.655985 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.656832 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.657833 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.659460 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.660085 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.660208 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.663304 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.663309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.663313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.663344 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.666109 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.666827 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.666831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.666834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.666956 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.667277 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.667282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.667285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.667661 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.667785 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.668115 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.668445 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":43}
I20260927 22:52:37.668833 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.655352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45962)
0927 22:52:37.655602 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.655620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.655720 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.656024 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.656029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.656478 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.656679 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.656691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.657971 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.657977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.659288 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.659305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.660239 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.660246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.660468 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.660991 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.661007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.663143 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.666244 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.666249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.666252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.666703 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.667062 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.667066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.667070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.667186 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.667761 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.667766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.667896 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.668020 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.668308 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1}
I20260927 22:52:37.674512 22984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: Bootstrap starting.
I20260927 22:52:37.678407 22984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:37.679800 22984 log.cc:824] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:52:37.684803 22984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: No bootstrap required, opened a new log
I20260927 22:52:37.690289 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.671336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.0.97.61:42651)
0927 22:52:37.671522 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.671528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.671544 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:37.671603 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.671608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.671680 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.671763 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.673376 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.674420 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.676139 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.676757 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.676870 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.682267 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.682276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.682282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.682333 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.685816 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.686631 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.686637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.686640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.686787 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.687251 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.687256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.687259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.687613 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.687870 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.689742 (+ 1872us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.689956 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:52:37.694749 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:37.694870 22984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.695241 22984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470efaa4d8934b93a29ff19ab3afed70, State: Initialized, Role: FOLLOWER
I20260927 22:52:37.695799 22984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.698410 22811 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "00236005e51440f098eb56e9310e6d6e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44790
I20260927 22:52:37.698854 22811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:52:37.699071 22811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:37.689971 22769 master_runner.cc:186] Error getting master registration for 127.0.97.61:42651: OK, instance_id { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" instance_seqno: 1790549557132888 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260927 22:52:37.702123 23003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:52:37.702237 23003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:52:37.702386 22769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:52:37.703074 22811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.703006 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.703522 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:37.704252 22984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:37.705029 22811 leader_election.cc:288] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [CANDIDATE]: Term 1 election: Requested vote from peers b5b0d427e07b4b61bd5a41810cbb7221 (127.0.97.61:42651), 470efaa4d8934b93a29ff19ab3afed70 (127.0.97.60:41431)
I20260927 22:52:37.705283 22959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00236005e51440f098eb56e9310e6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "470efaa4d8934b93a29ff19ab3afed70"
I20260927 22:52:37.705797 22959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:37.708999 22959 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00236005e51440f098eb56e9310e6d6e in term 1.
I20260927 22:52:37.709729 22785 leader_election.cc:302] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00236005e51440f098eb56e9310e6d6e, 470efaa4d8934b93a29ff19ab3afed70; no voters:
I20260927 22:52:37.710193 22991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:37.723241 22991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [term 1 LEADER]: Becoming Leader. State: Replica: 00236005e51440f098eb56e9310e6d6e, State: Running, Role: LEADER
I20260927 22:52:37.724002 22991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.731666 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00236005e51440f098eb56e9310e6d6e. Latest consensus state: current_term: 1 leader_uuid: "00236005e51440f098eb56e9310e6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.732189 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:37.740151 23019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:37.742466 23019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:37.754715 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.719341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.97.62:46409)
0927 22:52:37.719534 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.719540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.729143 (+ 9603us) server_negotiation.cc:407] Connection header received
0927 22:52:37.729653 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.729659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.729751 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.729832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.732151 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.733331 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.737979 (+ 4648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.738785 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.738887 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.746155 (+ 7268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.746161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.746166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.746204 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.750460 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.751565 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.751570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.751574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.751747 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.752463 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.752468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.752471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.752860 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.753116 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.754102 (+ 986us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.754362 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:52:37.755491 22959 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [term 1 FOLLOWER]: Refusing update from remote peer 00236005e51440f098eb56e9310e6d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:37.756867 23008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:37.782332 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00236005e51440f098eb56e9310e6d6e. Latest consensus state: current_term: 1 leader_uuid: "00236005e51440f098eb56e9310e6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.782668 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:37.789978 23022 mvcc.cc:205] Tried to move back new op lower bound from 7334090988553768960 to 7334090988444061696. Current Snapshot: MvccSnapshot[applied={T|T < 7334090988553768960}]
I20260927 22:52:37.791668 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.718215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.0.97.60:41431)
0927 22:52:37.718396 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.718403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.747233 (+ 28830us) server_negotiation.cc:407] Connection header received
0927 22:52:37.747655 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.747661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.747754 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.747826 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.754058 (+ 6232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.755141 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.764125 (+ 8984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.764873 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.764976 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.776166 (+ 11190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.776174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.776179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.776221 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.780072 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.786437 (+ 6365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.786443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.786447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.786672 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.788158 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.788164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.788167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.788551 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.788789 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.791015 (+ 2226us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.791284 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:52:37.792964 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.719807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.97.61:42651)
0927 22:52:37.719999 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.720006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.749602 (+ 29596us) server_negotiation.cc:407] Connection header received
0927 22:52:37.750228 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.750235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.750320 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.750391 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.759287 (+ 8896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.760372 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.767348 (+ 6976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.768028 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.768132 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.781011 (+ 12879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.781018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.781023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.781068 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.784631 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.787407 (+ 2776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.787413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.787417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.787589 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.789508 (+ 1919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.789514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.789517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.789885 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.790105 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.792379 (+ 2274us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.792600 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:52:37.795701 23025 mvcc.cc:205] Tried to move back new op lower bound from 7334090988553768960 to 7334090988444061696. Current Snapshot: MvccSnapshot[applied={T|T < 7334090988553768960}]
I20260927 22:52:37.799460 23019 catalog_manager.cc:1347] Generated new cluster ID: ee676af4a9514164b361c7cbe6d6cac1
I20260927 22:52:37.799573 23019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:37.815716 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00236005e51440f098eb56e9310e6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.816038 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:37.805140 22917 master_runner.cc:186] Error getting master registration for 127.0.97.61:42651: OK, instance_id { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" instance_seqno: 1790549557132888 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:52:37.818949 22917 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:52:37.827945 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00236005e51440f098eb56e9310e6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.828188 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:37.834458 23030 catalog_manager.cc:1259] Loaded cluster ID: ee676af4a9514164b361c7cbe6d6cac1
I20260927 22:52:37.834537 23030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: loading cluster ID for follower catalog manager: success
W20260927 22:52:37.835640 23030 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260927 22:52:37.835772 23030 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:52:37.849858 23019 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:37.852450 23019 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:37.852582 22910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } attempt: 5
Sep 27 22:52:37 dist-test-slave-kr25 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:52:37.876309 23019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e: Generated new TSK 0
I20260927 22:52:37.877442 23019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:37.880021 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.853482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41431 (local address 127.0.0.1:45984)
0927 22:52:37.853805 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:37.853825 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:37.853930 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:37.854902 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:37.854907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:37.855416 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:37.855657 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.855670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.857347 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.857352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.858738 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:37.858754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.860266 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.860272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:37.860477 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.861221 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:37.861262 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:37.868213 (+ 6951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:37.875549 (+ 7336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.875555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.875559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.876565 (+ 1006us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.877391 (+ 826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:37.877397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:37.877401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:37.877788 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:37.879006 (+ 1218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:37.879011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:37.879171 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 22:52:37.879309 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:52:37.879621 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260927 22:52:37.881052 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.854332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.0.97.60:41431)
0927 22:52:37.854490 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.854496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.854512 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:37.854572 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.854577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.854645 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.854733 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.856029 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.857144 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.858935 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.859883 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.860007 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.868496 (+ 8489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.868503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.868508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.868556 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.875315 (+ 6759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.876762 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.876770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.876774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.877120 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.877963 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.877969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.878012 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.878702 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.878894 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.880460 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.880685 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:52:37.887120 22910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221: Bootstrap starting.
I20260927 22:52:37.892503 22910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:37.893814 22910 log.cc:824] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:37.904448 22910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221: No bootstrap required, opened a new log
I20260927 22:52:37.914489 22910 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.914906 22910 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b0d427e07b4b61bd5a41810cbb7221, State: Initialized, Role: FOLLOWER
I20260927 22:52:37.915423 22910 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } }
I20260927 22:52:37.936870 23035 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.937642 23035 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:37.938639 22885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00236005e51440f098eb56e9310e6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b5b0d427e07b4b61bd5a41810cbb7221"
I20260927 22:52:37.938999 22884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:37.942653 22884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [term 1 FOLLOWER]: Refusing update from remote peer 00236005e51440f098eb56e9310e6d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:37.943742 22910 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:37.946143 22885 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00236005e51440f098eb56e9310e6d6e in term 1.
I20260927 22:52:37.947449 22991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:37.983572 23035 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00236005e51440f098eb56e9310e6d6e. Latest consensus state: current_term: 1 leader_uuid: "00236005e51440f098eb56e9310e6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.983899 23035 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:37.984330 23043 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00236005e51440f098eb56e9310e6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:37.984774 23043 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:37.998669 23053 catalog_manager.cc:1259] Loaded cluster ID: ee676af4a9514164b361c7cbe6d6cac1
I20260927 22:52:37.999251 23053 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221: loading cluster ID for follower catalog manager: success
I20260927 22:52:38.004185 23053 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221: acquiring CA information for follower catalog manager: success
I20260927 22:52:38.006819 23053 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5b0d427e07b4b61bd5a41810cbb7221: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:52:38.045106 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.010587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.0.97.61:42651)
0927 22:52:38.010932 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.010937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.042765 (+ 31828us) server_negotiation.cc:407] Connection header received
0927 22:52:38.043403 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.043411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.043534 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.043622 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.044842 (+ 1220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37796: BlockingRecv error: recv got EOF from 127.0.0.1:37796 (error 108)
Metrics: {"server-negotiator.queue_time_us":219}
W20260927 22:52:38.047216 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.010746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.0.97.60:41431)
0927 22:52:38.011073 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.011079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.042186 (+ 31107us) server_negotiation.cc:407] Connection header received
0927 22:52:38.045668 (+ 3482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.045674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.045762 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.045837 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.046952 (+ 1115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46000: BlockingRecv error: recv got EOF from 127.0.0.1:46000 (error 108)
Metrics: {"server-negotiator.queue_time_us":209}
W20260927 22:52:38.049996 22839 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.010415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.0.97.62:46409)
0927 22:52:38.010622 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.010629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.048022 (+ 37393us) server_negotiation.cc:407] Connection header received
0927 22:52:38.048654 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.048660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.048769 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.048846 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.049753 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44818: BlockingRecv error: recv got EOF from 127.0.0.1:44818 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:52:38.119575 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.103875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.0.97.61:42651)
0927 22:52:38.104073 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.104079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.111073 (+ 6994us) server_negotiation.cc:407] Connection header received
0927 22:52:38.112054 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.112063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.112311 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.112494 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.113255 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.114401 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.116422 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.117064 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.117169 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.117982 (+ 813us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.118729 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.118848 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.119101 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.119195 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:52:38.165870 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.157047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.0.97.60:41431)
0927 22:52:38.157251 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.157257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.163980 (+ 6723us) server_negotiation.cc:407] Connection header received
0927 22:52:38.164315 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.164322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.164454 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.164537 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.165634 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46012: BlockingRecv error: recv got EOF from 127.0.0.1:46012 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:52:38.234670 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.217212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.0.97.61:42651)
0927 22:52:38.217421 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.217428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.228052 (+ 10624us) server_negotiation.cc:407] Connection header received
0927 22:52:38.228269 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.228275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.228364 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.228427 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.229119 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.230446 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.232363 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.233056 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.233152 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.233795 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.233960 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.234023 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.234278 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.234363 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 22:52:38.256366 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.254197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.97.60:41431)
0927 22:52:38.254393 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.254399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.254898 (+ 499us) server_negotiation.cc:407] Connection header received
0927 22:52:38.255083 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.255089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.255187 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.255309 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.256121 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46022: BlockingRecv error: recv got EOF from 127.0.0.1:46022 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:52:38.268345 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.261509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.0.97.61:42651)
0927 22:52:38.261680 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.261686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.262152 (+ 466us) server_negotiation.cc:407] Connection header received
0927 22:52:38.262337 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.262343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.262474 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.262566 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.263012 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.264176 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.266152 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.266823 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.266942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.267380 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.267608 (+ 228us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.267696 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.267835 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.267923 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:52:38.281400 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.274615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.0.97.60:41431)
0927 22:52:38.274779 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.274785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.279998 (+ 5213us) server_negotiation.cc:407] Connection header received
0927 22:52:38.280171 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.280176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.280267 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.280374 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.281164 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46032: BlockingRecv error: recv got EOF from 127.0.0.1:46032 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:52:38.295553 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.288892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.0.97.61:42651)
0927 22:52:38.289129 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.289134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.289614 (+ 480us) server_negotiation.cc:407] Connection header received
0927 22:52:38.289825 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.289832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.289936 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.290027 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.290484 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.291601 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.293515 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.294237 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.294362 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.294813 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.294942 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.295040 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.295159 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.295239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 22:52:38.304661 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.302417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.0.97.60:41431)
0927 22:52:38.302691 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.302697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.303219 (+ 522us) server_negotiation.cc:407] Connection header received
0927 22:52:38.303404 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.303410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.303509 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.303607 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.304420 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46042: BlockingRecv error: recv got EOF from 127.0.0.1:46042 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
I20260927 22:52:38.322775 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.315937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.0.97.61:42651)
0927 22:52:38.316138 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.316146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.316682 (+ 536us) server_negotiation.cc:407] Connection header received
0927 22:52:38.316884 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.316890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.316982 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.317094 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.317582 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.318731 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.320651 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.321367 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.321501 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.321981 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.322128 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.322213 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.322379 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.322469 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 22:52:38.333382 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.329252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.97.60:41431)
0927 22:52:38.329448 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.329454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.331963 (+ 2509us) server_negotiation.cc:407] Connection header received
0927 22:52:38.332141 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.332148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.332263 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.332372 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.333168 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46058: BlockingRecv error: recv got EOF from 127.0.0.1:46058 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:38.359829 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.352940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.97.61:42651)
0927 22:52:38.353154 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.353160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.353700 (+ 540us) server_negotiation.cc:407] Connection header received
0927 22:52:38.353898 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.353904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.353996 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.354096 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.354561 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.355767 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.357690 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.358319 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.358483 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.358992 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.359163 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.359253 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.359464 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.359552 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 22:52:38.368196 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.365996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.0.97.60:41431)
0927 22:52:38.366176 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.366182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.366680 (+ 498us) server_negotiation.cc:407] Connection header received
0927 22:52:38.366854 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.366860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.366960 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.367079 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.367927 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46068: BlockingRecv error: recv got EOF from 127.0.0.1:46068 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:52:38.411302 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.404623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.0.97.61:42651)
0927 22:52:38.404832 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.404841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.405332 (+ 491us) server_negotiation.cc:407] Connection header received
0927 22:52:38.405597 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.405604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.405701 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.405811 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.406276 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.407364 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.409323 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.409972 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.410099 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.410533 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.410677 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.410770 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.410894 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.410974 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 22:52:38.423322 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.421208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.0.97.60:41431)
0927 22:52:38.421381 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.421386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.421879 (+ 493us) server_negotiation.cc:407] Connection header received
0927 22:52:38.422057 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.422063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.422176 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.422289 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.423087 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46082: BlockingRecv error: recv got EOF from 127.0.0.1:46082 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:52:38.495039 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.490494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.0.97.61:42651)
0927 22:52:38.490713 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.490719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.490983 (+ 264us) server_negotiation.cc:407] Connection header received
0927 22:52:38.491159 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.491164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.491234 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.491316 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.491709 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.492410 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.493655 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.494029 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.494143 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.494442 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.494546 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.494608 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.494720 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.494780 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:52:38.505853 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.504276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.97.60:41431)
0927 22:52:38.504428 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.504432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.504723 (+ 291us) server_negotiation.cc:407] Connection header received
0927 22:52:38.504891 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.504895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.504964 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.505062 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.505648 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46094: BlockingRecv error: recv got EOF from 127.0.0.1:46094 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:52:38.544508 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00236005e51440f098eb56e9310e6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00236005e51440f098eb56e9310e6d6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46409 } } peers { permanent_uuid: "b5b0d427e07b4b61bd5a41810cbb7221" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42651 } } peers { permanent_uuid: "470efaa4d8934b93a29ff19ab3afed70" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41431 } } }
I20260927 22:52:38.544860 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00236005e51440f098eb56e9310e6d6e [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:38.641525 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.636931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.0.97.61:42651)
0927 22:52:38.637113 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.637118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.637465 (+ 347us) server_negotiation.cc:407] Connection header received
0927 22:52:38.637637 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.637642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.637718 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.637814 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.638121 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.638819 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.640088 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.640484 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.640570 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.640955 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.641082 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.641154 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.641260 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.641318 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 22:52:38.648299 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:38.646548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.97.60:41431)
0927 22:52:38.646742 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.646748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.646985 (+ 237us) server_negotiation.cc:407] Connection header received
0927 22:52:38.647169 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.647175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.647262 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.647349 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.648078 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46096: BlockingRecv error: recv got EOF from 127.0.0.1:46096 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:52:38.839100 23030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: acquiring CA information for follower catalog manager: success
I20260927 22:52:38.840895 23030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 470efaa4d8934b93a29ff19ab3afed70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:52:38.913630 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.908712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.0.97.61:42651)
0927 22:52:38.908913 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.908919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.909195 (+ 276us) server_negotiation.cc:407] Connection header received
0927 22:52:38.909398 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.909403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.909498 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.909605 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.910032 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.910735 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.912057 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.912471 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.912592 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.912976 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.913083 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.913157 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.913298 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.913379 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:52:38.923761 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.918988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.0.97.60:41431)
0927 22:52:38.919151 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.919156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.919366 (+ 210us) server_negotiation.cc:407] Connection header received
0927 22:52:38.919526 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.919530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.919678 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.919768 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.920094 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.920742 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.921898 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.922287 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.922381 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.922774 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.923305 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.923415 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.923485 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.923557 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:38.935890 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.929134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.0.97.61:42651)
0927 22:52:38.929318 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.929324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.930232 (+ 908us) server_negotiation.cc:407] Connection header received
0927 22:52:38.930408 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.930413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.930485 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.930574 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.931167 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.932076 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.933876 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.934446 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.934562 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.935028 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.935171 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.935245 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.935516 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.935588 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:52:38.936338 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.929994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.0.97.60:41431)
0927 22:52:38.930164 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.930170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.930513 (+ 343us) server_negotiation.cc:407] Connection header received
0927 22:52:38.930834 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.930839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.930922 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.931007 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.931537 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.932597 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.934343 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.934871 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.934979 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.935367 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.935526 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.935626 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.935764 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.935840 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:52:38.937055 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.929760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.0.97.62:46409)
0927 22:52:38.931081 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.931085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.931102 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:52:38.931218 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.931223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.931414 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.931617 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:38.932087 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.932876 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.934720 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.935293 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.935406 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.935933 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:38.936409 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:38.936512 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.936704 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.936784 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":1133,"threads_started":1}
I20260927 22:52:38.942502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22769
I20260927 22:52:38.943497 22831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:39.051033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22769
I20260927 22:52:39.068203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22843
I20260927 22:52:39.069247 22905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:39.173787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22843
I20260927 22:52:39.189731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 22917
I20260927 22:52:39.190768 22979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:39.293989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 22917
2026-09-27T22:52:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4290 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:39 dist-test-slave-kr25 krb5kdc[23177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:39 dist-test-slave-kr25 krb5kdc[23177](info): set up 2 sockets
Sep 27 22:52:39 dist-test-slave-kr25 krb5kdc[23177](info): commencing operation
krb5kdc: starting...
W20260927 22:52:40.303727 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:40 dist-test-slave-kr25 krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:40.542977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37727
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37727,127.0.97.61:36033,127.0.97.60:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:40.857491 23190 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:52:40.857789 23190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:40.857859 23190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:40.857939 23190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:40.864984 23190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:40.865067 23190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:40.865128 23190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:40.865175 23190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:40.865232 23190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:40.874687 23190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37727,127.0.97.61:36033,127.0.97.60:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:40.875883 23190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:40.877099 23190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:40.887754 23195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:40.887800 23198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:40.887784 23196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:40.888406 23190 server_base.cc:1034] running on GCE node
I20260927 22:52:40.889112 23190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:40.890352 23190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:40.891521 23190 hybrid_clock.cc:648] HybridClock initialized: now 1790549560891456 us; error 39 us; skew 500 ppm
I20260927 22:52:40.891811 23190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:40 dist-test-slave-kr25 krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:40.898264 23190 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:40.899667 23190 webserver.cc:460] Webserver started at http://127.0.97.62:43261/ using document root <none> and password file <none>
I20260927 22:52:40.900347 23190 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:40.900445 23190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:40.900736 23190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:40.903687 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data/instance:
uuid: "7a1eafb5cc9f4cf087039dd3101cbabe"
format_stamp: "Formatted at 2026-09-27 22:52:40 on dist-test-slave-kr25"
server_key: "c84c6c86984c8a2b52f7274d7295f0ee"
server_key_iv: "3ccbf45bfeaff7094b1a70937555a668"
server_key_version: "encryptionkey@0"
I20260927 22:52:40.904421 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/wal/instance:
uuid: "7a1eafb5cc9f4cf087039dd3101cbabe"
format_stamp: "Formatted at 2026-09-27 22:52:40 on dist-test-slave-kr25"
server_key: "c84c6c86984c8a2b52f7274d7295f0ee"
server_key_iv: "3ccbf45bfeaff7094b1a70937555a668"
server_key_version: "encryptionkey@0"
I20260927 22:52:40.909235 23190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:52:40.912474 23205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:40.913746 23190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:40.913908 23190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/wal
uuid: "7a1eafb5cc9f4cf087039dd3101cbabe"
format_stamp: "Formatted at 2026-09-27 22:52:40 on dist-test-slave-kr25"
server_key: "c84c6c86984c8a2b52f7274d7295f0ee"
server_key_iv: "3ccbf45bfeaff7094b1a70937555a668"
server_key_version: "encryptionkey@0"
I20260927 22:52:40.914101 23190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:40.940142 23190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:40.943783 23190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:40.944018 23190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:40.960147 23190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37727
I20260927 22:52:40.960191 23256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37727 every 8 connection(s)
I20260927 22:52:40.961382 23190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:40.964080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 23190
I20260927 22:52:40.964254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:40.964797 388 external_mini_cluster.cc:1428] Setting key e26646acb266a00178dd0d6758bfdac4
I20260927 22:52:40.965580 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:40.973698 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:52:40 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 22:52:40.995055 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.977214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.97.62:37727)
0927 22:52:40.977936 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:40.977946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:40.978472 (+ 526us) server_negotiation.cc:407] Connection header received
0927 22:52:40.980403 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:40.980408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:40.980639 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:40.980817 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:40.982010 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:40.983170 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:40.984523 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:40.985049 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:40.985218 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:40.988501 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:40.988509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:40.988520 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:40.988560 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:40.991152 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:40.991882 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:40.991889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:40.991891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:40.992033 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:40.992447 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:40.992451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:40.992453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:40.992909 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:40.993064 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:40.993721 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 22:52:40.993978 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":536,"thread_start_us":331,"threads_started":1}
I20260927 22:52:40.995080 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.977215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37727 (local address 127.0.0.1:37420)
0927 22:52:40.978323 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:40.978359 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:40.980060 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:40.980838 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:40.980844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:40.981506 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:40.981826 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:40.981842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:40.983304 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:40.983308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:40.984400 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:40.984413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:40.985257 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:40.985264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:40.985427 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:40.985989 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:40.986010 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:40.988287 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:40.991280 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:40.991290 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:40.991299 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:40.991751 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:40.992151 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:40.992155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:40.992158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:40.992353 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:40.993038 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:40.993046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:40.993309 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 22:52:40.993572 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:52:40.994159 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":114,"thread_start_us":761,"threads_started":1}
I20260927 22:52:40.998159 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:40.999181 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.998761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41608)
0927 22:52:40.998914 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:40.999040 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:52:40.999462 23206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
I20260927 22:52:41.001010 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.000623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41614)
0927 22:52:41.000765 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.000857 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:52:41.001211 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36033: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:52:41.022590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36033
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37727,127.0.97.61:36033,127.0.97.60:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:52:41.050541 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } attempt: 1
I20260927 22:52:41.051713 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.051264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41624)
0927 22:52:41.051475 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.051573 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:52:41.052652 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.052332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41640)
0927 22:52:41.052441 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.052502 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 22:52:41.052826 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36033: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
I20260927 22:52:41.121052 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } attempt: 2
I20260927 22:52:41.122160 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.121725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41652)
0927 22:52:41.121909 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.121999 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:52:41.123098 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.122774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41654)
0927 22:52:41.122882 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.122957 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 22:52:41.123279 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36033: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
I20260927 22:52:41.214515 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } attempt: 3
I20260927 22:52:41.215647 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.215216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41666)
0927 22:52:41.215400 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.215510 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:52:41.216625 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.216307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41682)
0927 22:52:41.216420 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.216481 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 22:52:41.216813 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36033: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
W20260927 22:52:41.335064 23264 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:52:41.335353 23264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:41.335448 23264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:41.335531 23264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:41.342519 23264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:41.342602 23264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:41.342664 23264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:41.342712 23264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:41.342769 23264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:41.352650 23264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37727,127.0.97.61:36033,127.0.97.60:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:41.353933 23264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:41.355211 23264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:52:41.360042 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } attempt: 4
I20260927 22:52:41.361367 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.360704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41692)
0927 22:52:41.361101 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.361203 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 22:52:41.362418 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.362027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41694)
0927 22:52:41.362169 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.362272 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:52:41.362617 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36033: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36033: connect: Connection refused (error 111)
W20260927 22:52:41.366775 23272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:41.366911 23269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:41.366866 23270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:41.367879 23264 server_base.cc:1034] running on GCE node
I20260927 22:52:41.368629 23264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:41.369959 23264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:41.371168 23264 hybrid_clock.cc:648] HybridClock initialized: now 1790549561371105 us; error 44 us; skew 500 ppm
I20260927 22:52:41.371515 23264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:41.374496 23264 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:52:41.375846 23264 webserver.cc:460] Webserver started at http://127.0.97.61:34749/ using document root <none> and password file <none>
I20260927 22:52:41.376475 23264 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:41.376556 23264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:41.376808 23264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:41.379590 23264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data/instance:
uuid: "c5aad2de9b8640e5ae8e54cb98933564"
format_stamp: "Formatted at 2026-09-27 22:52:41 on dist-test-slave-kr25"
server_key: "3c919fd1ea0717d9e504188d91be8261"
server_key_iv: "1a0feb6d5c5d2cfb9ad81ecd405aa7eb"
server_key_version: "encryptionkey@0"
I20260927 22:52:41.380288 23264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/wal/instance:
uuid: "c5aad2de9b8640e5ae8e54cb98933564"
format_stamp: "Formatted at 2026-09-27 22:52:41 on dist-test-slave-kr25"
server_key: "3c919fd1ea0717d9e504188d91be8261"
server_key_iv: "1a0feb6d5c5d2cfb9ad81ecd405aa7eb"
server_key_version: "encryptionkey@0"
I20260927 22:52:41.384949 23264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:52:41.388104 23279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:41.389374 23264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:41.389513 23264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/wal
uuid: "c5aad2de9b8640e5ae8e54cb98933564"
format_stamp: "Formatted at 2026-09-27 22:52:41 on dist-test-slave-kr25"
server_key: "3c919fd1ea0717d9e504188d91be8261"
server_key_iv: "1a0feb6d5c5d2cfb9ad81ecd405aa7eb"
server_key_version: "encryptionkey@0"
I20260927 22:52:41.389703 23264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:41.412328 23264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:41.416464 23264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:41.416705 23264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:41.432830 23264 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36033
I20260927 22:52:41.432854 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36033 every 8 connection(s)
I20260927 22:52:41.434146 23264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:52:41.437721 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:41.444175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 23264
I20260927 22:52:41.444340 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-1/wal/instance
I20260927 22:52:41.443912 23331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:41.444748 388 external_mini_cluster.cc:1428] Setting key 16bbb5fbc02d3df3cf2e32a7bb94a84b
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:41.463704 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.446424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.0.97.62:37727)
0927 22:52:41.446620 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.446627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.447798 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 22:52:41.449590 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.449596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.449677 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.449784 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.451144 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.452049 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.454079 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.454482 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.454568 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.458305 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.458310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.458313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.458347 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.460833 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.461607 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.461610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.461612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.461720 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.462095 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.462098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.462100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.462369 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.462482 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.463221 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.463406 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:52:41.465082 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.446857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37727 (local address 127.0.0.1:37436)
0927 22:52:41.447648 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.447696 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:41.449341 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:41.449964 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:41.449971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:41.450450 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:41.450988 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.451004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.452195 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.452201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.453948 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.453963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.454694 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.454701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.454963 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.455658 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:41.455685 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:41.458086 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:41.460957 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.460966 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.460981 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.461507 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.461805 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.461810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.461813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.462009 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.462481 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:41.462489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:41.462890 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 22:52:41.463165 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 22:52:41.463698 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":326,"threads_started":1}
I20260927 22:52:41.466856 23331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:52:41.480149 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.467643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41704)
0927 22:52:41.467796 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.467812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:41.467894 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:41.468677 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:41.468683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:41.469045 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:41.469171 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.469179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.470567 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.470572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.471364 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.471373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.472181 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.472186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.472342 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.472854 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:41.472870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:41.474875 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:41.477846 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.477852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.477856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.478201 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.478544 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.478549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.478553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.478663 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.479191 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:41.479196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:41.479363 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 22:52:41.479496 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:52:41.479768 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:52:41.480228 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.467737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.0.97.61:36033)
0927 22:52:41.468271 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.468277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.468299 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:52:41.468337 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.468342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.468505 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.468636 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.469316 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.470431 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.471501 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.472053 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.472159 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.475082 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.475093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.475106 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.475134 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.477727 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.478313 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.478320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.478322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.478428 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.478751 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.478756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.478760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.479084 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.479275 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.479645 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.479866 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260927 22:52:41.482697 23331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:41.483718 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.483254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45920)
0927 22:52:41.483426 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.483572 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:52:41.483964 23280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
I20260927 22:52:41.485394 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.485103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45922)
0927 22:52:41.485226 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.485299 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:52:41.485596 23331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:52:41.503554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
/tmp/dist-test-taskfHsC0g/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35419
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37727,127.0.97.61:36033,127.0.97.60:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:52:41.534857 23331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 1
I20260927 22:52:41.536101 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.535613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45932)
0927 22:52:41.535825 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.535919 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:52:41.537334 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.536944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45936)
0927 22:52:41.537103 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.537211 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 22:52:41.537581 23331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
I20260927 22:52:41.605798 23331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 2
I20260927 22:52:41.606897 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.606472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45942)
0927 22:52:41.606664 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.606752 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:52:41.607827 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.607521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45950)
0927 22:52:41.607655 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.607718 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 22:52:41.608018 23331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
I20260927 22:52:41.661854 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } attempt: 5
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:52:41.679107 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.662484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41720)
0927 22:52:41.662732 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.662755 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:41.662884 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:41.663236 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:41.663241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:41.663798 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:41.664063 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.664077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.665530 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.665536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.667216 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.667235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.668696 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.668703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.668917 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.669594 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:41.669616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:41.672289 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:41.676204 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.676210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.676214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.676762 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.677243 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.677248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.677251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.677409 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.678112 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:41.678117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:41.678296 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 22:52:41.678445 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:52:41.678737 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 22:52:41.680063 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.662706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.0.97.61:36033)
0927 22:52:41.662861 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.662865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.662884 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:52:41.663002 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.663005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.663082 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.663163 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.664255 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.665322 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.667472 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.668201 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.668341 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.672509 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.672516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.672521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.672581 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.676036 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.676926 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.676933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.676936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.677103 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.677536 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.677541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.677556 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.677911 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.678050 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.679476 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.679751 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:52:41.681348 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:41.682682 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.682137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45962)
0927 22:52:41.682420 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.682509 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260927 22:52:41.683701 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.683284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45964)
0927 22:52:41.683461 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.683563 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 22:52:41.684240 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
I20260927 22:52:41.699203 23331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 3
I20260927 22:52:41.700160 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.699756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45972)
0927 22:52:41.699940 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.700020 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:52:41.701157 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.700775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45982)
0927 22:52:41.700982 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.701054 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260927 22:52:41.701354 23331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
I20260927 22:52:41.735481 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 1
I20260927 22:52:41.736578 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.736137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:45988)
0927 22:52:41.736319 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.736404 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:52:41.737465 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.737191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46004)
0927 22:52:41.737303 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.737366 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 22:52:41.737689 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
I20260927 22:52:41.805867 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 2
I20260927 22:52:41.806951 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.806511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46006)
0927 22:52:41.806717 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.806809 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:52:41.807883 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.807594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46008)
0927 22:52:41.807712 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.807775 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 22:52:41.808074 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
W20260927 22:52:41.823112 23338 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:52:41.823410 23338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:41.823526 23338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:41.823639 23338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:41.830600 23338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:41.830685 23338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:41.830770 23338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:41.830830 23338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:41.830912 23338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:41.840425 23338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37727,127.0.97.61:36033,127.0.97.60:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:41.841636 23338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:41.842850 23338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:52:41.844534 23331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 4
I20260927 22:52:41.845495 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.845069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46010)
0927 22:52:41.845277 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.845357 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:52:41.846427 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.846136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46024)
0927 22:52:41.846265 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.846330 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:52:41.846627 23331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
W20260927 22:52:41.853466 23346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:41.853502 23343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:41.853430 23344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:41.854454 23338 server_base.cc:1034] running on GCE node
I20260927 22:52:41.855136 23338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:41.856412 23338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:41.857623 23338 hybrid_clock.cc:648] HybridClock initialized: now 1790549561857559 us; error 32 us; skew 500 ppm
I20260927 22:52:41.857915 23338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:41.860749 23338 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:52:41.862020 23338 webserver.cc:460] Webserver started at http://127.0.97.60:33761/ using document root <none> and password file <none>
I20260927 22:52:41.862679 23338 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:41.862776 23338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:41.863058 23338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:41.865903 23338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data/instance:
uuid: "8e446475a9914ff78ec25b02ca1de3f4"
format_stamp: "Formatted at 2026-09-27 22:52:41 on dist-test-slave-kr25"
server_key: "9c0c5917515aba130637ab54693398b6"
server_key_iv: "4e968d022de35b6db95b7a470cab4ea1"
server_key_version: "encryptionkey@0"
I20260927 22:52:41.866719 23338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/wal/instance:
uuid: "8e446475a9914ff78ec25b02ca1de3f4"
format_stamp: "Formatted at 2026-09-27 22:52:41 on dist-test-slave-kr25"
server_key: "9c0c5917515aba130637ab54693398b6"
server_key_iv: "4e968d022de35b6db95b7a470cab4ea1"
server_key_version: "encryptionkey@0"
I20260927 22:52:41.871667 23338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 22:52:41.875039 23353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:41.876410 23338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:41.876585 23338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/wal
uuid: "8e446475a9914ff78ec25b02ca1de3f4"
format_stamp: "Formatted at 2026-09-27 22:52:41 on dist-test-slave-kr25"
server_key: "9c0c5917515aba130637ab54693398b6"
server_key_iv: "4e968d022de35b6db95b7a470cab4ea1"
server_key_version: "encryptionkey@0"
I20260927 22:52:41.876784 23338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:41.906435 23338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:41.910055 23338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:41.910293 23338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:41.914278 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 3
I20260927 22:52:41.915426 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.914921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46026)
0927 22:52:41.915168 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.915267 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 22:52:41.916522 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.916182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46034)
0927 22:52:41.916340 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.916411 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:52:41.916771 23257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35419: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35419: connect: Connection refused (error 111)
I20260927 22:52:41.927011 23338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35419
I20260927 22:52:41.927058 23404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35419 every 8 connection(s)
I20260927 22:52:41.928333 23338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:52:41.931831 23405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:41.937353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu as pid 23338
I20260927 22:52:41.937505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHsC0g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549216186161-388-0/minicluster-data/master-2/wal/instance
I20260927 22:52:41.937882 388 external_mini_cluster.cc:1428] Setting key b626733d7b7090392c1d817e4319b29c
I20260927 22:52:41.937979 23405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:52:41.955825 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.940672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.0.97.62:37727)
0927 22:52:41.940858 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.940864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.941312 (+ 448us) server_negotiation.cc:407] Connection header received
0927 22:52:41.942440 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.942446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.942516 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.942598 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.943840 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.944517 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.945921 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.946374 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.946498 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.950105 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.950110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.950114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.950147 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.952746 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.953661 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.953664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.953667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.953785 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.954191 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.954195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.954197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.954453 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.954592 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.955219 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.955466 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:52:41.956776 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.940562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37727 (local address 127.0.0.1:37442)
0927 22:52:41.941168 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.941211 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:41.942213 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:41.942678 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:41.942684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:41.943283 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:41.943680 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.943698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.944663 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.944667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.945779 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.945794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.946580 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.946585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.946841 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.947505 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:41.947540 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:41.949929 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:41.952873 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.952886 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.952898 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.953549 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.953896 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.953901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.953905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.954076 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.954583 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:41.954591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:41.954877 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 22:52:41.955120 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 22:52:41.955599 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":42,"thread_start_us":268,"threads_started":1}
I20260927 22:52:41.956794 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.939526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.0.97.60:35419)
0927 22:52:41.940236 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.940249 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.940294 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:52:41.942206 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.942225 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.942451 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.942627 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.944015 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.945159 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.946376 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.947010 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.947106 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.950834 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.950843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.950854 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.950884 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.953774 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.954382 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.954388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.954390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.954502 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.954833 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.954836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.954838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.955197 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.955485 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.955785 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.956006 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":506,"spinlock_wait_cycles":14592,"thread_start_us":302,"threads_started":1}
I20260927 22:52:41.958536 23405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:41.972584 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.959218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36033 (local address 127.0.0.1:41728)
0927 22:52:41.959366 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.959382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:41.959502 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:41.960060 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:41.960065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:41.960391 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:41.960566 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.960578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.961948 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.961954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.963060 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.963072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.963753 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.963758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.963937 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.964975 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:41.964990 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:41.967052 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:41.970485 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.970489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.970492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.970926 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.971253 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.971258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.971262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.971426 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.971883 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:41.971887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:41.972030 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 22:52:41.972147 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:52:41.972323 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:52:41.972908 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.959268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.0.97.61:36033)
0927 22:52:41.959553 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.959559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.959578 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:52:41.959652 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.959656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.959733 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.959826 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.960722 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.961804 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.963185 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.963630 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.963739 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.967212 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.967217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.967220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.967250 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.970367 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.971027 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.971030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.971033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.971154 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.971547 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.971550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.971552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.971780 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.971882 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.972268 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.972469 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 22:52:41.973806 23405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:52:41.974459 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.960017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.97.62:37727)
0927 22:52:41.960216 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.960222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.960237 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:52:41.960280 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.960284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.960346 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.960452 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.961557 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.962606 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.964284 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.964701 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.964791 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.969293 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.969298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.969301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.969329 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.971875 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.972492 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.972497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.972500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.972653 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.973092 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.973096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.973098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.973390 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.973661 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.973960 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.974178 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:52:41 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:52:41.986102 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.974441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46060)
0927 22:52:41.974605 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:41.974623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:41.974717 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:41.975233 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:41.975237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:41.975702 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:41.975858 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.975870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.977359 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.977363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.978190 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:41.978200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.978793 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.978798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:41.978972 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.979490 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:41.979509 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:41.981446 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:41.984005 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.984009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.984013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.984336 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.984637 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:41.984642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:41.984646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:41.984761 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:41.985190 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:41.985195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:41.985318 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 22:52:41.985441 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:52:41.985695 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:52:41.986141 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.974814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.0.97.60:35419)
0927 22:52:41.974956 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.974961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.974977 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:41.975017 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.975022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.975097 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.975192 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.976262 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.977246 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.978309 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.978675 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.978774 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.981605 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.981609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.981612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.981637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.983903 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.984431 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.984436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.984439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.984550 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.984847 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.984852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.984856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.985092 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.985206 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.985530 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.985839 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":44}
I20260927 22:52:41.988780 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:41.976147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.0.97.61:36033)
0927 22:52:41.976286 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:41.976290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:41.976303 (+ 13us) server_negotiation.cc:407] Connection header received
0927 22:52:41.976337 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:41.976340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:41.976387 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:41.976453 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:41.977741 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.978402 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.979757 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:41.980182 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:41.980292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:41.984082 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:41.984086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:41.984089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:41.984116 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:41.986452 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.987025 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.987030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.987041 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.987174 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:41.987505 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:41.987510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:41.987513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:41.987852 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:41.988128 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:41.988275 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:52:41.988478 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:52:41.991101 23405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: Bootstrap starting.
I20260927 22:52:41.992971 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:52:41.994303 23405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:41.995352 23405 log.cc:824] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:41.997855 23405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: No bootstrap required, opened a new log
I20260927 22:52:41.995236 23232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37458
I20260927 22:52:42.004882 23405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.005170 23405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e446475a9914ff78ec25b02ca1de3f4, State: Initialized, Role: FOLLOWER
I20260927 22:52:42.005580 23405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.006400 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.006767 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:42.007238 23405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:52:42.015302 23422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:52:42.015398 23422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:52:42.013748 23338 master_runner.cc:186] Error getting master registration for 127.0.97.62:37727: OK, instance_id { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" instance_seqno: 1790549560955486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:52:42.024438 23338 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:52:42.094025 23257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 4
Sep 27 22:52:42 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:52:42.109781 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:42.094785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46076)
0927 22:52:42.095001 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:42.095023 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:42.095185 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:42.095553 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:42.095558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:42.096097 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:42.096303 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:42.096315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.097721 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.097727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:42.099207 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:42.099224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.100215 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.100222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:42.100427 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:42.101075 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:42.101099 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:42.103911 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:42.107007 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:42.107012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:42.107016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:42.107402 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:42.107863 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:42.107868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:42.107872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:42.108036 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:42.108744 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:42.108749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:42.108940 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 22:52:42.109089 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:52:42.109391 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:52:42.109827 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:42.094986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.0.97.60:35419)
0927 22:52:42.095163 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.095168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.095190 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:52:42.095307 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.095312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.095391 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.095529 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.096453 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.097571 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.099352 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.100064 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.100183 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:42.104085 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:42.104092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:42.104097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:42.104137 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:42.106883 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.107549 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.107554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.107558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.107754 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.108150 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.108156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.108159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.108616 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:42.108784 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:42.109234 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 22:52:42.109462 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:52:42.115684 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe: Bootstrap starting.
I20260927 22:52:42.119684 23257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:42.120950 23257 log.cc:824] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:42.124167 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe: No bootstrap required, opened a new log
I20260927 22:52:42.131480 23257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.131855 23257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1eafb5cc9f4cf087039dd3101cbabe, State: Initialized, Role: FOLLOWER
I20260927 22:52:42.132335 23257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.133236 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.133626 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:42.134438 23257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:42.134537 23232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:52:42.134716 23232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:42.137562 23232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.139952 23232 leader_election.cc:288] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [CANDIDATE]: Term 1 election: Requested vote from peers c5aad2de9b8640e5ae8e54cb98933564 (127.0.97.61:36033), 8e446475a9914ff78ec25b02ca1de3f4 (127.0.97.60:35419)
I20260927 22:52:42.140069 23380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e446475a9914ff78ec25b02ca1de3f4"
I20260927 22:52:42.140537 23380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:42.143996 23380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1eafb5cc9f4cf087039dd3101cbabe in term 1.
I20260927 22:52:42.144588 23206 leader_election.cc:302] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a1eafb5cc9f4cf087039dd3101cbabe, 8e446475a9914ff78ec25b02ca1de3f4; no voters:
I20260927 22:52:42.145067 23425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:42.145638 23436 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:52:42.145805 23331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } attempt: 5
Sep 27 22:52:42 dist-test-slave-kr25 krb5kdc[23177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:52:42.157392 23425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [term 1 LEADER]: Becoming Leader. State: Replica: 7a1eafb5cc9f4cf087039dd3101cbabe, State: Running, Role: LEADER
I20260927 22:52:42.158723 23425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.162999 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:42.146333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35419 (local address 127.0.0.1:46084)
0927 22:52:42.146670 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:42.146689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:42.146795 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:42.147628 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:42.147634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:42.148119 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:42.148319 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:42.148331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.149762 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.149768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:42.151216 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:42.151231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.152484 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.152491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:42.152701 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:42.153375 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:42.153400 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:42.156158 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:42.160168 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:42.160174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:42.160179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:42.160685 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:42.161119 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:42.161124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:42.161128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:42.161310 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:42.161950 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:42.161955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:42.162163 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 22:52:42.162323 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:52:42.162613 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
W20260927 22:52:42.163409 23359 tablet.cc:2369] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:52:42.165102 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:42.146986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.97.60:35419)
0927 22:52:42.147161 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.147166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.147186 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:52:42.147237 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.147243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.147330 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.147418 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.148483 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.149607 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.151388 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.152156 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.152273 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:42.156363 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:42.156370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:42.156375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:42.156417 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:42.159993 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.160806 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.160811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.160814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.160985 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.161418 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.161424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.161441 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.161838 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:42.162001 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:42.164491 (+ 2490us) server_negotiation.cc:299] Negotiation successful
0927 22:52:42.164728 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:42.170943 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: Bootstrap starting.
I20260927 22:52:42.174935 23331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:42.176267 23331 log.cc:824] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:42.180650 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: No bootstrap required, opened a new log
I20260927 22:52:42.187891 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1eafb5cc9f4cf087039dd3101cbabe. Latest consensus state: current_term: 1 leader_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.188364 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:42.179476 23190 master_runner.cc:186] Error getting master registration for 127.0.97.61:36033: OK, instance_id { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" instance_seqno: 1790549561428320 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:52:42.190798 23190 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 22:52:42.196622 23211 tablet.cc:2369] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:52:42.197835 23331 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.198575 23331 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5aad2de9b8640e5ae8e54cb98933564, State: Initialized, Role: FOLLOWER
I20260927 22:52:42.199615 23331 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } }
I20260927 22:52:42.200341 23444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:42.201045 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.201512 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:42.202265 23331 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:42.202339 23306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c5aad2de9b8640e5ae8e54cb98933564"
I20260927 22:52:42.202839 23306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:42.206696 23306 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1eafb5cc9f4cf087039dd3101cbabe in term 1.
W20260927 22:52:42.234066 23458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:52:42.234265 23458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:52:42.234756 23444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:42.262025 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:42.201067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.97.61:36033)
0927 22:52:42.201253 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.201259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.217445 (+ 16186us) server_negotiation.cc:407] Connection header received
0927 22:52:42.217647 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.217651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.217726 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.217816 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.221992 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.223113 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.234091 (+ 10978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.234805 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.234914 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:42.251609 (+ 16695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:42.251616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:42.251622 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:42.251664 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:42.255513 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.257695 (+ 2182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.257701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.257705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.257864 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.259547 (+ 1683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.259553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.259557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.259931 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:42.260181 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:42.261368 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0927 22:52:42.261633 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:52:42.283162 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:42.203542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.0.97.62:37727)
0927 22:52:42.203701 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.203706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.218779 (+ 15073us) server_negotiation.cc:407] Connection header received
0927 22:52:42.218934 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.218939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.219012 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.219099 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.246495 (+ 27396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.247686 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.258326 (+ 10640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.259072 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.259176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:42.269393 (+ 10217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:42.269401 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:42.269407 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:42.269457 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:42.273207 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.278655 (+ 5448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.278662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.278665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.278844 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.279320 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.279325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.279329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.279753 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:42.280012 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:42.282531 (+ 2519us) server_negotiation.cc:299] Negotiation successful
0927 22:52:42.282784 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:42.284962 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:42.219337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.97.60:35419)
0927 22:52:42.219544 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.219550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.220435 (+ 885us) server_negotiation.cc:407] Connection header received
0927 22:52:42.220806 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.220812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.220903 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.220979 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.249852 (+ 28873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.250954 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.265459 (+ 14505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:42.266173 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:42.266279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:42.274110 (+ 7831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:42.274137 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:42.274142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:42.274186 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:42.277826 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.280206 (+ 2380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.280212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.280215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.280461 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:42.281366 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:42.281371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:42.281375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:42.281746 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:42.281993 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:42.284357 (+ 2364us) server_negotiation.cc:299] Negotiation successful
0927 22:52:42.284617 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:52:42.293164 23306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [term 1 FOLLOWER]: Refusing update from remote peer 7a1eafb5cc9f4cf087039dd3101cbabe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:42.293262 23380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [term 1 FOLLOWER]: Refusing update from remote peer 7a1eafb5cc9f4cf087039dd3101cbabe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:42.294234 23425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:42.294575 23438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:42.312738 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1eafb5cc9f4cf087039dd3101cbabe. Latest consensus state: current_term: 1 leader_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.313099 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:42.320335 23462 mvcc.cc:205] Tried to move back new op lower bound from 7334091007141957632 to 7334091006607593472. Current Snapshot: MvccSnapshot[applied={T|T < 7334091007141957632}]
I20260927 22:52:42.322535 23464 mvcc.cc:205] Tried to move back new op lower bound from 7334091007141957632 to 7334091006607593472. Current Snapshot: MvccSnapshot[applied={T|T < 7334091007141957632}]
I20260927 22:52:42.348955 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.349282 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:42.350185 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.350502 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:42.354274 23444 catalog_manager.cc:1347] Generated new cluster ID: e64dfc602d6244b0a0c8756d097db152
I20260927 22:52:42.354391 23444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:42.359352 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1eafb5cc9f4cf087039dd3101cbabe. Latest consensus state: current_term: 1 leader_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.359764 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:42.370646 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.371505 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:42.387588 23444 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:42.389436 23444 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:42.400098 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1eafb5cc9f4cf087039dd3101cbabe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37727 } } peers { permanent_uuid: "c5aad2de9b8640e5ae8e54cb98933564" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36033 } } peers { permanent_uuid: "8e446475a9914ff78ec25b02ca1de3f4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35419 } } }
I20260927 22:52:42.400380 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:42.415652 23444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a1eafb5cc9f4cf087039dd3101cbabe: Generated new TSK 0
I20260927 22:52:42.416675 23444 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:52:42.533403 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.517793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.0.97.61:36033)
0927 22:52:42.518015 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.518022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.529549 (+ 11527us) server_negotiation.cc:407] Connection header received
0927 22:52:42.531831 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.531838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.531951 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.532030 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.533143 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41754: BlockingRecv error: recv got EOF from 127.0.0.1:41754 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:52:42.537158 23260 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.526258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.0.97.62:37727)
0927 22:52:42.526455 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.526461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.534754 (+ 8293us) server_negotiation.cc:407] Connection header received
0927 22:52:42.535776 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.535781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.535886 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.535972 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.536939 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37476: BlockingRecv error: recv got EOF from 127.0.0.1:37476 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:52:42.539081 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.530058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.97.60:35419)
0927 22:52:42.530215 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.530220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.535336 (+ 5116us) server_negotiation.cc:407] Connection header received
0927 22:52:42.537784 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.537789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.537862 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.537936 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.538869 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46108: BlockingRecv error: recv got EOF from 127.0.0.1:46108 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 22:52:42.556167 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.554147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.0.97.61:36033)
0927 22:52:42.554341 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.554348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.554774 (+ 426us) server_negotiation.cc:407] Connection header received
0927 22:52:42.554980 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.554984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.555077 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.555161 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.555902 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41756: BlockingRecv error: recv got EOF from 127.0.0.1:41756 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:52:42.561292 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.559665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.0.97.61:36033)
0927 22:52:42.559826 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.559831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.560252 (+ 421us) server_negotiation.cc:407] Connection header received
0927 22:52:42.560389 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.560394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.560462 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.560545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.561140 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41772: BlockingRecv error: recv got EOF from 127.0.0.1:41772 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 22:52:42.570681 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.569156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.0.97.61:36033)
0927 22:52:42.569308 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.569314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.569436 (+ 122us) server_negotiation.cc:407] Connection header received
0927 22:52:42.569648 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.569652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.569732 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.569811 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.570463 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41780: BlockingRecv error: recv got EOF from 127.0.0.1:41780 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:52:42.579314 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.577520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.0.97.61:36033)
0927 22:52:42.577684 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.577690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.578136 (+ 446us) server_negotiation.cc:407] Connection header received
0927 22:52:42.578303 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.578307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.578373 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.578461 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.579082 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41786: BlockingRecv error: recv got EOF from 127.0.0.1:41786 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:52:42.591681 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.589879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.0.97.61:36033)
0927 22:52:42.590067 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.590073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.590500 (+ 427us) server_negotiation.cc:407] Connection header received
0927 22:52:42.590668 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.590673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.590735 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.590816 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.591467 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41790: BlockingRecv error: recv got EOF from 127.0.0.1:41790 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 22:52:42.612862 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.610827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.0.97.61:36033)
0927 22:52:42.611019 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.611026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.611519 (+ 493us) server_negotiation.cc:407] Connection header received
0927 22:52:42.611727 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.611731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.611803 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.611887 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.612605 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41792: BlockingRecv error: recv got EOF from 127.0.0.1:41792 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 22:52:42.650085 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.647956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.0.97.61:36033)
0927 22:52:42.648160 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.648168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.648675 (+ 507us) server_negotiation.cc:407] Connection header received
0927 22:52:42.648879 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.648883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.648964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.649055 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.649816 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41804: BlockingRecv error: recv got EOF from 127.0.0.1:41804 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:52:42.719395 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.717191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.97.61:36033)
0927 22:52:42.717434 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.717441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.718012 (+ 571us) server_negotiation.cc:407] Connection header received
0927 22:52:42.718208 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.718212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.718285 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.718382 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.719183 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41806: BlockingRecv error: recv got EOF from 127.0.0.1:41806 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:52:42.852536 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:42.850724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.0.97.61:36033)
0927 22:52:42.850932 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:42.850938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:42.851162 (+ 224us) server_negotiation.cc:407] Connection header received
0927 22:52:42.851353 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:42.851357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:42.851457 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:42.851542 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:42.852280 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41822: BlockingRecv error: recv got EOF from 127.0.0.1:41822 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:52:43.017284 23422 catalog_manager.cc:1259] Loaded cluster ID: e64dfc602d6244b0a0c8756d097db152
I20260927 22:52:43.017385 23422 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: loading cluster ID for follower catalog manager: success
I20260927 22:52:43.019884 23422 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: acquiring CA information for follower catalog manager: success
I20260927 22:52:43.021505 23422 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e446475a9914ff78ec25b02ca1de3f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:52:43.113545 23335 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:43.111757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.97.61:36033)
0927 22:52:43.111969 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:43.111976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:43.112166 (+ 190us) server_negotiation.cc:407] Connection header received
0927 22:52:43.112365 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:43.112369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:43.112442 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:43.112531 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:43.113284 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41830: BlockingRecv error: recv got EOF from 127.0.0.1:41830 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:52:43.236199 23458 catalog_manager.cc:1259] Loaded cluster ID: e64dfc602d6244b0a0c8756d097db152
I20260927 22:52:43.236316 23458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: loading cluster ID for follower catalog manager: success
I20260927 22:52:43.238890 23458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: acquiring CA information for follower catalog manager: success
I20260927 22:52:43.240617 23458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5aad2de9b8640e5ae8e54cb98933564: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:52:43.634032 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:43.628776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.97.61:36033)
0927 22:52:43.629242 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:43.629247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:43.629267 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:52:43.629425 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:43.629429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:43.629591 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:43.629709 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:43.630042 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.630752 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.632052 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.632444 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.632549 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:43.632989 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:43.633512 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:43.633622 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:43.633704 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 22:52:43.633773 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":243,"threads_started":1}
I20260927 22:52:43.644809 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:43.639344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.97.60:35419)
0927 22:52:43.639831 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:43.639836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:43.640083 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:52:43.640250 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:43.640254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:43.640395 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:43.640549 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:43.640830 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.641471 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.642719 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.643141 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.643251 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:43.643697 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:43.644230 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:43.644346 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:43.644436 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 22:52:43.644512 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260927 22:52:43.657234 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:43.650225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.0.97.61:36033)
0927 22:52:43.650424 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:43.650431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:43.652002 (+ 1571us) server_negotiation.cc:407] Connection header received
0927 22:52:43.652264 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:43.652269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:43.652349 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:43.652428 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:43.653038 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.653905 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.655458 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.655909 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.656021 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:43.656466 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:43.656609 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:43.656711 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:43.656837 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 22:52:43.656923 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:52:43.657931 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:43.651270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.0.97.60:35419)
0927 22:52:43.651462 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:43.651468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:43.652211 (+ 743us) server_negotiation.cc:407] Connection header received
0927 22:52:43.652482 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:43.652488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:43.652574 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:43.652656 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:43.653267 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.654074 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.656122 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.656673 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.656779 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:43.657263 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:43.657363 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:43.657441 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:43.657632 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 22:52:43.657704 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:52:43.658802 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:43.650223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.97.62:37727)
0927 22:52:43.650702 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:43.650709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:43.651319 (+ 610us) server_negotiation.cc:407] Connection header received
0927 22:52:43.651474 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:43.651481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:43.651714 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:43.651930 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:43.652638 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.653556 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.655804 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:43.656403 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:43.656530 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:43.657098 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:43.657525 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:43.657639 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:43.658433 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 22:52:43.658515 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":259,"threads_started":1}
I20260927 22:52:43.663594 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 23190
I20260927 22:52:43.664422 23252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:43.771029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 23190
I20260927 22:52:43.787425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 23264
I20260927 22:52:43.788496 23326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:43.891721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 23264
I20260927 22:52:43.907771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu pid 23338
I20260927 22:52:43.908797 23400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:44.012490 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHsC0g/build/asan/bin/kudu with pid 23338
2026-09-27T22:52:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4718 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9009 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (347782 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST